acceptodds
Under review as a conference paper at ICLR 2027

Token-Level Video Reinforcement Learning

Abstract

Reinforcement learning (RL) for video generation usually assigns one scalar reward to an entire sampled video. Yet a video is not uniformly flawed: some visual tokens may already satisfy the prompt, whereas others require correction. A scalar reward cannot localize errors, causing optimization to perturb satisfactory tokens while under-targeting the tokens that actually need to change. We introduce Token-Level Video Reinforcement Learning, TVRL, a framework that derives token-level credit from the reward being optimized. Our key insight is that the answer likelihood of a frozen vision-language model provides both signals: its outputs contribute to the video-level reward, while magnitudes of its video-input gradients reveal which generated video tokens most affect that score. We instantiate TVRL in Group Relative Policy Optimization by averaging prompt-derived question rewards into one group-relative advantage and using detached, question-conditioned token-credit maps to reweight dense denoising-transition log-probabilities inside the clipped policy ratio. On VBench-2.0, TVRL achieves an Overall score of 57.69, outperforming the base model by 3.60 points. TVRL also improves matched GRPO baselines across three SDE samplers (SAGE, Flow, and Dance) by 2.68–3.15 points and across four reward models (VideoAlign, VideoScore2, UnifiedReward2, and Qwen3.5-9B) by 1.33–3.15 points.

Primary area:
generative models
Keywords:
RL for video generation model, Video GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HUVWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TVRL uses frozen reward-model input gradients to distribute a video-level GRPO advantage across spatial and temporal units. It improves VBench-2.0 over matched GRPO baselines across several samplers and reward models.

Critical strengths
  1. Reward-derived routing addresses a relevant limitation of scalar-feedback video optimization.

  2. Multi-seed experiments across three samplers and four reward models, together with blind human evaluation, support the empirical gains.

  3. Ablations examine routing granularity, DINOv2-based maps, smoothing, and alternative ratio formulations.

Critical weaknesses
  1. Spatial grounding remains weakly established. Appendix A.6 assigns only 52.7% of credit to the queried panel versus 50% under uniform routing. The shuffled control also changes the final map statistics, leaving semantic alignment incompletely isolated.

  2. Shared advantages limit check-specific correction. A failed check can receive positive reinforcement when the overall rollout advantage is positive. The implications deserve stronger evaluation, particularly since controllability barely improves over matched GRPO.

  3. Compute efficiency is unclear. The reported 18% overhead comes from imperfectly matched historical VideoAlign runs, rather than the default Qwen3.5-9B configuration. Comparisons at equal optimizer steps do not establish equal-compute benefits.

Questions for authors
  1. Does routing remain beneficial against controls that preserve final-map smoothness and entropy while disrupting semantic correspondence?

  2. Can you quantify outcomes for failed checks under positive shared advantages, and provide a controlled advantage ablation clarifying the historical variant in Table A3?

  3. What are the default configuration’s GPU-hours, peak memory, and performance at matched wall-clock budgets?

4 comments

We thank the reviewer for the positive assessment and for three concrete requests, which we answer with new measurements below.

On seeds. All new training runs in this response use a single seed because of the time and compute available during the rebuttal; results from the submission are means over three seeds. We will report three seeds for every new run in the final version. We treat the single-seed results as preliminary evidence and do not draw conclusions from small numerical differences. The tables labelled R are new and will be added to the revised PDF.

W1 and Q1. Spatial grounding; controls that preserve map smoothness and entropy

The A.6 test measures the question–position interaction (D = +5.39 pp, p = 0.0034, 10 of 12 groups), which is zero for any score-independent map. We agree that the absolute mass shift is modest in this constructed 2×4 setting. We add two cleaner tests.

Distribution-preserving control (Q1). We reuse real maps unchanged, so smoothness, entropy, and the weight distribution are identical by construction and only the correspondence to the video is broken (Table R6). With maps from a video of a different prompt, Overall falls from 57.69 to 54.55, the level of uniform GRPO (54.54). With maps from another rollout of the same prompt and check, Overall is 56.72. Routing therefore remains beneficial only when the maps match the prompt and its checks; much of the signal is conditioned on the prompt and the check rather than on the individual rollout.

Table R6. Map controls (SAGE, Qwen3.5-9B, 100 steps). The two swap rows are new and single-seed; the other rows are from the paper.

Routing mapSame map statistics as TVRLMap taken fromOverall
Uniform (GRPO)––54.54
3×3 shuffled (paper)noown video, shuffled53.70
DINOv2 mapnoown video (perceptual)55.48
Cross-prompt swap (new)yesa video of another prompt54.55
Group swap (new)yesanother rollout, same prompt and check56.72
TVRLyesown video57.69

Question-swap test on generated videos. With the video and the object mask fixed, an object receives more credit when the question is about it than when it is about another object in the same video (Table R5). The effect is modest, in line with A.6: it supports question-conditioned sensitivity, not precise object-level localization.

Table R5. Question-swap test on generated videos with two segmentable objects (GroundingDINO + SAM masks). We ask "Does the video show a/an …?" once per object and measure the credit mass on a fixed object mask. Training critic configuration, 3×3 routing, coordinates as in Appendix A.6; lift = credit mass ÷ mask area, averaged over videos (1.0 = uniform). 12 videos pass a mask check fixed before any gradient was computed.

Question askedCredit mass on the object's maskLift
About this object13.31%1.14×
About the other object in the same video11.67%1.04×

Difference: +1.65 pp (95% CI [+0.43, +3.27]; exact paired sign-flip test, p = 0.0034; positive in 10 of 12 videos). A.6 reports the same p-value because both tests have 12 units and, in both, 14 of the 4,096 sign assignments reach the observed mean. On all 20 attempted videos, without the mask check: +1.28 pp, positive in 18 of 20.

W2. Shared advantages limit check-specific correction

As stated in Sec. 3.3, this is a deliberate choice, and we now evaluate it. We tested the alternative of a separate advantage per check: with the same frame-level credit maps it yields 54.42 Overall, vs. 56.47 with the shared advantage and 54.54 for uniform GRPO (Table R3, in part 2 under Q2).

The shared advantage is the average of the per-check centered scores, so each check contributes to the shared reward and supplies its own routing map, while the sign of the update is shared across checks. The per-check variant instead gives each check its own update direction and magnitude. It also normalizes every check separately over only 4 rollouts, so checks on which the rollouts barely differ receive the same weight as checks on which they clearly differ. The shared advantage normalizes once, after pooling the 5 checks.

Controllability. We agree that it barely moves on HunyuanVideo-1.5 (31.57 → 31.64), where the gains concentrate in Creativity (+5.68) and Physics (+8.63). On a second backbone, Wan2.2-TI2V-5B, it improves from 21.63 (GRPO) to 26.58 (TVRL), with Overall 53.24 → 56.34 in a single-seed run, which suggests that the flat result is specific to that setting. We report the HunyuanVideo-1.5 result as it is and will note it in the limitations.

Q2. Failed checks under positive shared advantages; controlled ablation for Table A3

(1) Outcomes of failed checks. Among the checks that the base model fails on held-out prompts, TVRL fixes 45.5% vs. 25.5% for GRPO, and it breaks 5.5% of passing checks vs. 8.2% for GRPO (Table R4). A failed check lowers the reward of its rollout, so it receives a positive advantage only when the other checks of that rollout keep its reward above the group mean. Both methods train with the same shared advantage, so the rates above include the effect of all such updates; their net outcome is that failed checks are fixed more often, and more so with token routing. This is an aggregate measure: it does not isolate the individual rollouts in which a failed check received a positive advantage.

(2) Controlled ablation for Table A3. The "Token credit" variant uses frame-level credit with a separate advantage per check. Its controlled counterpart is the Frame row of Table 3 (same frame-level credit, shared advantage), which differs only in the advantage: 54.42 vs. 56.47 (Table R3). We will rewrite the caption to say so in the revised PDF.

Table R4. Failed checks on held-out prompts: 200 prompts × 3 seeds × 5 checks = 3,000 checks, of which 825 fail under the base model (p(Yes) < 0.5). Step-100 checkpoints, same seeds for all models. The scorer is the training critic, so we compare the two methods and do not interpret the absolute rates.

ModelCheck pass rateFix rate (base fail → pass)Break rate (base pass → fail)
Base72.5%––
GRPO73.6%25.5%8.2%
TVRL81.0%45.5%5.5%

Table R3. Shared vs. per-check advantage. The second and third rows differ only in the advantage.

VariantAdvantageCredit granularityOverall
GRPO (uniform)sharednone54.54
"Token credit" (Table A3)per checkframe54.42
Frame credit (Table 3)sharedframe56.47
TVRLshared3×357.69

W3 and Q3. Compute for the default configuration; matched wall-clock

We agree that the VideoAlign pair does not measure the default configuration. Table R8 reports the default Qwen3.5-9B configuration from the logged training runs (64 A100 80GB GPUs, per-device batch size 2; mean time per update over updates 11–100). GRPO takes 710.5 s per update and TVRL 844.5 s (+18.9%), i.e. 1,263 vs. 1,501 GPU-hours per 100 updates, with an observed device-level peak memory of 54.6 vs. 77.9 GiB. The higher peak memory, close to the 80 GB capacity of the device, is a practical cost of the method.

The policy-optimization stage takes the same time in both runs (420 s). About 12 s of the difference falls in the rollout stage, where the critic's backward pass runs, and the rest comes from recomputing old-policy log-probabilities for weighted routing. Measured in isolation on one GPU, computing the credit maps for 5 questions raises the critic's time from 0.80 to 3.33 s per video and its peak memory from 18.1 to 21.0 GiB.

Matched wall-clock. In the time TVRL needs for 100 updates, GRPO can run 119. We did not evaluate GRPO at exactly 119 updates, so the following comparison is indicative, not exactly matched. Two results suggest that the extra updates would not close the gap. First, GRPO does not improve beyond 100 updates in our long-run experiment (54.54 at step 100, 53.34 at step 200, 51.54 at step 300; Table R1 in our response to Reviewer 8o94). Second, TVRL at step 50, which costs 59% of the wall-clock of GRPO's 100 updates, already reaches 56.74.

Table R8. Compute for the default Qwen3.5-9B configuration (SAGE, seed 42; 64 A100 80GB GPUs). Peak memory is the largest sampled device-level allocation on the reporting host.

Methods per updateRelativeGPU-hours per 100 updatesPeak GPU memory (observed)
GRPO710.51.00×1,26354.6 GiB
TVRL844.51.19×1,50177.9 GiB

Thanks for your deliberate rebuttal, most of my concern have been resolved. I maintain my recommend for acceptance.

Thank you for your thoughtful feedback and for recognizing our efforts to address your concerns. Your suggestions helped us strengthen the experimental evidence and clarify the architectural contributions. We appreciate your time and constructive discussion.

Reviewer_NbpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TVRL uses a frozen VLM to score several questions derived from the prompt. The scores are averaged to give each video a GRPO advantage. The input gradient of each question's score with respect to the video then gives a spatiotemporal weight map. These maps weight the local denoising log-probabilities before the GRPO ratio is formed and clipped, and reward gradients do not pass through the generator. On HunyuanVideo-1.5, the default configuration raises the VBench-2.0 Overall score from 54.54 for the matched GRPO baseline to 57.69. The paper also reports results with different samplers, critics, and human preference.

Critical strengths
  1. Assigning different reward weights to different tokens is a valuable idea. Different regions and time steps of a video do not matter equally for a given requirement. Weighting token updates according to each question helps focus optimization on the relevant content, and is a useful complement to uniform video-level feedback.

  2. The results on the tested generator provide some support. On HunyuanVideo-1.5, improvements are reported across samplers and reward models, and are further supported by weight-map ablations and human preference results. This suggests that local weighting is worth further study.

Critical weaknesses
  1. Rewards and penalties are still determined by the video-level advantage; the token-level design mainly acts as regional weighting. The question scores are first averaged into one video score, which yields a single advantage shared by all questions and positions. The gradient magnitude only sets how much each local log-probability contributes to the update; it does not provide an independent local reward or penalty direction. As a result, even if a requirement is not met, the related regions may still be reinforced as long as the video as a whole is better than the other samples in its group. Regions with large gradients are also not necessarily the ones that need correction. In addition, since DiT parameters are shared, local weighting cannot guarantee that an update affects only the corresponding positions. This is not a flaw in the algorithm itself, but the current evidence is not yet sufficient to show that the method achieves the targeted correction of flawed regions emphasized in the introduction. I would encourage the authors to distinguish more clearly between local sensitivity weighting and local error correction, and to provide more direct validation.

  2. Effectiveness across generators and RL algorithms has not been verified. All experiments use a single generator backbone, HunyuanVideo-1.5. The SAGE, Flow, and Dance comparisons mainly change the stochastic denoising sampler under GRPO while keeping the same type of clipped group-relative objective, so they cannot substitute for testing a different RL objective. I suggest adding at least one other generator backbone and one different RL objective, to check whether the token weighting still consistently improves the corresponding baseline. Possible backbones include Wan2.2 TI2V-5B or T2V-A14B, LTX-2, or MiniMax-H3.

Questions for authors
  1. Given that all positions share the video-level advantage, could you further verify that the weight maps actually help correct unmet requirements? The existing shuffled control changes both the spatial structure and the final weight distribution, so it provides limited support for this mechanism.

  2. Could you add a comparison between this token weighting and the corresponding uniform-weight baseline, using another generator backbone and an RL objective other than the current GRPO?

  3. In Table A3, how exactly does the "Token credit" variant (54.42) differ from the final method (57.69) in terms of advantage, weight granularity, and ratio? Does the uniform-weight baseline use the same question-averaged reward and normalization?

  4. Please check the statistics. In Table 1, the five-dimension averages for VideoScore2/TVRL and UnifiedReward2/GRPO are 55.586 and 55.018, which do not match the reported 55.99 and 54.82. Also, if the Overall score of each seed is the average of the five dimensions, its standard deviation should not exceed the average of the five dimension standard deviations. For VideoAlign/TVRL in Table A1, however, the reported Overall standard deviation is 0.42, while that average is about 0.304. Please explain this and provide the corresponding per-seed scores.

5 comments

We thank the reviewer for the precise reading and for checking our numbers.

W1. The token-level design is regional weighting, not local error correction

We agree with this characterization. Sec. 3.3 states that all checks share one advantage and that "a failed check can therefore receive a positive shared advantage", but the abstract and Fig. 2 overstate this as local correction. In the revised PDF we will describe TVRL as reward-sensitivity weighting of a shared advantage, and add that shared DiT parameters make the update non-local. We also agree that the gradient magnitude measures sensitivity, not error: it selects where the update is concentrated, and the advantage sets the direction.

Can a failed requirement be reinforced? For a single rollout it can, as Sec. 3.3 states. We tested the alternative of a separate advantage per check: with the same frame-level credit maps it yields 54.42 Overall, vs. 56.47 with the shared advantage and 54.54 for uniform GRPO (Table R3, in part 3 under Q3). The shared advantage is the average of the per-check centered scores, so each check contributes to the shared reward and supplies its own routing map, while the sign of the update is shared. The per-check variant instead gives each check its own update direction and magnitude; it also normalizes every check separately over only 4 rollouts, so checks on which the rollouts barely differ receive the same weight as checks on which they clearly differ, whereas the shared advantage normalizes once, after pooling the 5 checks.

Direct validation. Among the checks that the base model fails on held-out prompts, TVRL fixes 45.5% vs. 25.5% for GRPO, with break rates of 5.5% vs. 8.2% (Table R4). On generated videos, an object receives more credit when the question is about it than when the question is about another object in the same video (lift 1.14 vs. 1.04, p = 0.0034; Table R5). This effect is modest but specific to the question.

Table R4. Failed checks on held-out prompts: 200 prompts × 3 seeds × 5 checks = 3,000 checks, of which 825 fail under the base model (p(Yes) < 0.5). Step-100 checkpoints, same seeds for all models. The scorer is the training critic, so we compare the two methods and do not interpret the absolute rates.

ModelCheck pass rateFix rate (base fail → pass)Break rate (base pass → fail)
Base72.5%––
GRPO73.6%25.5%8.2%
TVRL81.0%45.5%5.5%

Table R5. Question-swap test on generated videos with two segmentable objects (GroundingDINO + SAM masks). We ask "Does the video show a/an …?" once per object and measure the credit mass on a fixed object mask. Training critic configuration, 3×3 routing, coordinates as in Appendix A.6; lift = credit mass ÷ mask area, averaged over videos (1.0 = uniform). 12 videos pass a mask check fixed before any gradient was computed.

Question askedCredit mass on the object's maskLift
About this object13.31%1.14×
About the other object in the same video11.67%1.04×

Difference: +1.65 pp (95% CI [+0.43, +3.27]; exact paired sign-flip test, p = 0.0034; positive in 10 of 12 videos). A.6 reports the same p-value because both tests have 12 units and, in both, 14 of the 4,096 sign assignments reach the observed mean. On all 20 attempted videos, without the mask check: +1.28 pp, positive in 18 of 20.

W2 and Q2. Another generator backbone and another RL objective

Backbone. We added Wan2.2-TI2V-5B, one of the backbones the reviewer suggested. TVRL improves its matched uniform-weight GRPO baseline from 53.24 to 56.34 Overall (+3.10) and is higher than GRPO in all five dimensions (Table R9). This is one additional backbone, not evidence for every architecture. Creativity is below the base model for both methods on this backbone, less so for TVRL.

Table R9. Second backbone, Wan2.2-TI2V-5B (Qwen3.5-9B reward, 100 steps, single seed). The two GRPO rows differ only in the token weighting.

ModelOverallCreat.Comm.Ctrl.HumanPhys.
Base52.5953.5061.5719.5087.2041.19
GRPO, uniform weights53.2446.9162.8321.6389.7945.01
GRPO, TVRL weights56.3450.1166.6826.5890.8247.51

RL objective. We did not run a second objective, and we do not claim results beyond the policy-gradient family; we will state this in the limitations. As a conceptual observation only: the credit map is a per-token weight and is not tied to the GRPO form. In an objective such as DiffusionNFT, which weights velocity-regression losses by the reward, the map could allocate the reward-dependent factor over tokens in the same way. What does not transfer is our per-question clipped ratio, which is specific to clipped policy-gradient objectives. Whether the map helps under such an objective is untested.

Q1. Do the weight maps help, given that the shuffled control is confounded?

We agree that the shuffled control is confounded. We added two controls that reuse real maps unchanged, so smoothness, entropy, and the weight distribution are identical by construction and only the correspondence to the video is broken (Table R6). With maps taken from a video of a different prompt, Overall falls from 57.69 to 54.55, the level of uniform GRPO (54.54): the gain disappears. With maps taken from another rollout of the same prompt and check, Overall is 56.72: most of the gain is kept, so a large part of the signal is conditioned on the prompt and the check rather than specific to the individual rollout. The gain therefore depends on the maps matching the prompt and its checks, not on their statistics; we do not read it as evidence of exact error localization. Whether unmet requirements are corrected more often is answered by Table R4 (part 2, under W1).

Table R6. Map controls (SAGE, Qwen3.5-9B, 100 steps). The two swap rows are new and single-seed; the other rows are from the paper.

Routing mapSame map statistics as TVRLMap taken fromOverall
Uniform (GRPO)––54.54
3×3 shuffled (paper)noown video, shuffled53.70
DINOv2 mapnoown video (perceptual)55.48
Cross-prompt swap (new)yesa video of another prompt54.55
Group swap (new)yesanother rollout, same prompt and check56.72
TVRLyesown video57.69

Q3. How "Token credit" (54.42) in Table A3 differs from the final method (57.69)

"Token credit" differs from the final method in two respects.

  • Advantage: each question receives its own advantage, normalized by that question's mean and standard deviation over the 4 rollouts of the group and routed by its own map, instead of the shared advantage.
  • Granularity: frame-level credit instead of 3×3.

The ratio is the same per-question weighted ratio. Its matched counterpart is therefore the Frame row of Table 3 (frame-level credit with the shared advantage, 56.47), from which it differs only in the advantage, so the gap between 54.42 and 56.47 isolates the effect of the shared advantage (Table R3). The uniform baseline (54.54) uses the same question-averaged log-probability reward and group normalization as TVRL. We will rewrite the Table A3 caption accordingly in the revised PDF.

Table R3. Shared vs. per-check advantage. The second and third rows differ only in the advantage.

VariantAdvantageCredit granularityOverall
GRPO (uniform)sharednone54.54
"Token credit" (Table A3)per checkframe54.42
Frame credit (Table 3)sharedframe56.47
TVRLshared3×357.69

Q4. Statistics

You are correct on both points.

(1) The Overall of two rows of Table 1 was not the mean of the five dimensions. Recomputed from the reported dimension scores, VideoScore2/TVRL is 55.59 (not 55.99) and UnifiedReward2/GRPO is 55.02 (not 54.82). The range of gains over the four reward models becomes +0.93 to +3.15 instead of +1.33 to +3.15. TVRL remains above GRPO for every reward model. We will correct Table 1 and the abstract in the revised PDF.

Reward modelGRPOTVRLGain
VideoAlign54.1855.54+1.36
VideoScore254.6655.59 (was 55.99)+0.93 (was +1.33)
UnifiedReward255.02 (was 54.82)56.67+1.65 (was +1.85)
Qwen3.5-9B54.5457.69+3.15

(2) You are also right that the Overall standard deviations in Table A1 are inconsistent with the dimension standard deviations. Besides VideoAlign/TVRL, this affects VideoScore2/TVRL, UnifiedReward2/GRPO, and Qwen3.5-9B/GRPO. We are recomputing all standard deviations from the per-run evaluation outputs. The per-seed scores you ask for, together with the corrected Table A1, will be provided in the revised PDF.

On seeds. All new training runs in this response use a single seed because of the time and compute available during the rebuttal; results from the submission are means over three seeds. We will report three seeds for every new run in the final version. We treat the single-seed results as preliminary evidence and do not draw conclusions from small numerical differences. The tables labelled R are new and will be added to the revised PDF.

Reviewer_8o94OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Token-Level Video Reinforcement Learning (TVRL), a framework that aims to improve credit assignment in video generation by moving beyond a single reward for the entire video. The key idea is to use a frozen VLM to obtain both scalar rewards and token-level credit. Specifically, the authors use teacher-forced answer likelihoods to compute rewards for different questions about the generated video, while the magnitudes of the corresponding gradients provide spatio-temporal credit maps. This avoids the need for additional reward-model training or dense credit annotations. Building on GRPO, the method combines question-level rewards into a shared group-relative advantage and uses the credit maps to reweight denoising-transition log-probabilities within the clipped policy-ratio objective. This allows the model to focus its updates on reward-sensitive regions of the video rather than treating all tokens equally.

Critical strengths
  1. The paper tackles a very important limitation of video GRPO, where a single reward is assigned to all video tokens regardless of which regions actually need improvement.
  2. The proposed approach is simple and easy to use as it uses gradient from existing frozen VLMs for the token level credit assignment and doesnt require any additional reward model training or dense token-level labels.
  3. TVRL performs better than GRPO, across different reward models and different SDE sampling methods, showing the generality of there method.
Critical weaknesses
  1. The GRPO baselines are very weak, they barely improve over the pretrained model raising concerns about where they were sufficiently optimized. Since all the methods are evaluated after only 100 training steps, without demonstrating GRPO convergence or adequate hyperparameter tuning, it is unclear whether TVRL would maintain its gains against a fulling converged GRPO baseline.

  2. The proposed method assumes that larger reward gradients indicate tokens requiring greater updates. However, gradients can be misleading or noisy and the method can end up reward hacking. Since the experiments only evaluate models after 100 optimizer steps, it is difficult to assess whether the proposed credit assignment remains reliable over longer training runs or eventually leads to reward exploitation. Further analysis of training stability and reward hacking would strengthen the paper.

  3. The method doesnt compare with a close prior work ViPO [1] which also tries to address the problem of fine grained credit assignment by redistributing GRPO advantages using perceptual features. The authors only include a DINOv2-based routing ablation rather than comparing against the full ViPO method, which uses a different policy optimization objective with pixel-level advantages and local likelihood ratios. A comparison under matched experimental settings would help establish whether TVRL offers meaningful improvements over this closely related prior work.

  4. Although the method decomposes prompts into individual semantic questions, it aggregatestheir rewards into a single shared advantage. As a result, correctly and incorrectly generated regions can receive the same update direction, potentially reinforcing unsatisfied requirements. For example, a video that satisfies four out of five requirements may receive a positive overall advantage, even though one requirement remains incorrect. The paper does not investigate this failure mode or compare against question-specific advantage which would help justify the use of a shared advantage.

[1] Ni, Z., Liang, Y., Li, R., Zhou, Y., Huang, H., Zhang, C., & Li, X. (2026). Seeing What Matters: Visual Preference Policy Optimization for Visual Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Questions for authors

See weaknesses.

4 comments

We thank the reviewer for the careful reading and for recognizing the importance of the problem and the generality across samplers and reward models. We answer each point below.

On seeds. All new training runs in this response use a single seed because of the time and compute available during the rebuttal; results from the submission are means over three seeds. We will report three seeds for every new run in the final version. We treat the single-seed results as preliminary evidence and do not draw conclusions from small numerical differences. The tables labelled R are new and will be added to the revised PDF.

W1. Weak GRPO baselines; only 100 training steps

We follow the official SAGE-GRPO protocol, which trains for 100 steps. GRPO and TVRL use identical hyperparameters and differ only in the routing map. In this setting, training longer does not strengthen the baseline: both methods score lower at the later checkpoints we evaluated (Table R1). TVRL is higher than GRPO at every checkpoint (+2.42 to +4.78), and the best GRPO checkpoint (54.54, step 100) remains below every TVRL checkpoint (56.32 to 57.69). By step 300, GRPO (51.54) is below the base model (54.09).

The small gain of GRPO over the base model is consistent with weak transfer from the training reward to the independent VBench-2.0 benchmark. With the same reward and step budget, TVRL adds +0.93 to +3.15 on top of GRPO across the four reward models (range corrected, see our response to Reviewer NbpE, Q4). This inefficiency of trajectory-level credit is what TVRL targets.

Table R1. Long-run training (SAGE, Qwen3.5-9B reward), VBench-2.0 Overall. Step 100 is the 3-seed mean from the paper; the other steps are single-seed.

StepGRPOTVRLGap
5054.3256.74+2.42
10054.5457.69+3.15
20053.3457.46+4.12
30051.5456.32+4.78

W2. Noisy gradients and reward hacking over longer training

We trained both methods to 300 steps (Table R1). Beyond the official 100-step point, both GRPO and TVRL decline on VBench-2.0, so the decline is not specific to token routing. TVRL declines less (−1.37 vs. −3.00 between the step-100 and step-300 checkpoints) and remains above GRPO at every checkpoint. All reported metrics are disjoint from the training reward: VBench-2.0 uses its own evaluators, and the human study uses human raters. This is evidence that the gains generalize beyond the training critic; it does not by itself rule out reward exploitation. Gradient noise is reduced by spatial smoothing (the unsmoothed 1×1 variant reaches 55.10 vs. 57.69 for 3×3). Clipping constrains the weighted policy ratio of each question (Sec. 3.4); it does not bound each token's ratio individually.

W3. Comparison with the full ViPO method

We compare with a ViPO-style adaptation in our video training pipeline (HunyuanVideo-1.5, Qwen3.5-9B reward, SAGE, 100 steps), which incorporates ViPO's perceptual credit source and its local likelihood ratios under matched training conditions. This is not a reproduction of ViPO in its original training setting. To our knowledge, ViPO has no public code release, and it reports its video results on Wan2.1-T2V-14B with a VideoAlign reward, so its published numbers are not comparable with ours. We therefore re-implemented its ingredients inside our pipeline, where the backbone, reward, sampler, and step budget are held fixed; the perceptual map follows the construction of ViPO's Perceptual Structuring Module on DINOv2 features.

  1. Credit source. Replacing our reward-gradient map with ViPO's perceptual DINOv2 map gives 55.48 Overall, vs. 54.54 for GRPO and 57.69 for TVRL (Table 3).
  2. Pixel-level advantages. For a fixed map ww, a per-token advantage A^ wg\hat{A}\,w_g and our weighted log-probability have the same gradient at the old policy, before clipping. The two objectives then differ in the form of the ratio, which item 3 tests.
  3. Local likelihood ratios. Clipping local (frame-wise) ratios lowers Overall to 52.95, vs. 54.96 for a single ratio and 56.47 for our per-question ratios (Table A5).

New: combining the DINOv2 map with local ratios, i.e. the ViPO-style objective in our setting, gives 54.85 in a single-seed run (Table R2). This is numerically below the DINOv2 map alone (55.48), a single-seed difference that we do not interpret, and 2.84 below TVRL (57.69). These results indicate that deriving credit from the reward being optimized is a main source of TVRL's gain in our setting. They do not isolate every difference between the two objectives, and we make no claim about ViPO in its original setting.

Table R2. ViPO components in our setting. The combined row is new (single seed); the other rows are from the paper.

VariantCredit sourceRatioOverall
GRPOuniformsingle54.54
ViPO credit sourceDINOv2 map, 3×3weighted55.48
Local ratiosVLM gradient, frameper frame, clipped locally52.95
Single averaged ratioVLM gradient, framesingle54.96
Per-question ratiosVLM gradient, frameper question56.47
ViPO-style combined (new)DINOv2 map, 3×3local54.85
TVRLVLM gradient, 3×3per question57.69

W4. A shared advantage can reinforce an unmet requirement

As stated in Sec. 3.3, sharing one advantage across checks is a deliberate choice. We tested the alternative the reviewer suggests: with the same frame-level credit maps, giving each check its own advantage yields 54.42 Overall, vs. 56.47 with the shared advantage and 54.54 for uniform GRPO (Table R3).

The shared advantage is the average of the per-check centered scores, so each check contributes to the shared reward and supplies its own routing map, while the sign of the update is shared across checks. The per-check variant instead gives each check its own update direction and magnitude. It also normalizes every check separately over only 4 rollouts, so checks on which the rollouts barely differ receive the same weight as checks on which they clearly differ. The shared advantage normalizes once, after pooling the 5 checks.

On the failure mode itself: among the checks that the base model fails on held-out prompts, TVRL fixes 45.5% vs. 25.5% for GRPO, and it breaks fewer passing checks (5.5% vs. 8.2%; Table R4). This is an aggregate outcome: the shared advantage does not exclude that an unmet check is reinforced in an individual rollout, as Sec. 3.3 states.

Table R3. Shared vs. per-check advantage. The second and third rows differ only in the advantage.

VariantAdvantageCredit granularityOverall
GRPO (uniform)sharednone54.54
"Token credit" (Table A3)per checkframe54.42
Frame credit (Table 3)sharedframe56.47
TVRLshared3×357.69

Table R4. Failed checks on held-out prompts: 200 prompts × 3 seeds × 5 checks = 3,000 checks, of which 825 fail under the base model (p(Yes) < 0.5). Step-100 checkpoints, same seeds for all models. The scorer is the training critic, so we compare the two methods and do not interpret the absolute rates.

ModelCheck pass rateFix rate (base fail → pass)Break rate (base pass → fail)
Base72.5%––
GRPO73.6%25.5%8.2%
TVRL81.0%45.5%5.5%

I thank the authors for their detailed responses and additional experiments. I believe W1 and W4 have been sufficiently addressed. However, I still have some remaining concerns regarding W2 and W3.

W2: The authors do not directly investigate reward hacking or establish whether high-gradient regions actually correspond to semantic errors, particularly given the performance decline over longer training. It would be helpful to report training rewards alongside independent evaluation metrics over longer training runs and analyze whether high-gradient regions consistently correspond to semantic errors.

W3: The ViPO-style comparison is a useful addition, but it is based on a single seed and is not a validated reproduction of the full ViPO method. This makes it difficult to assess the reliability and fairness of the comparison. Evaluating the baseline across multiple seeds and providing further implementation details to establish its consistency with the original ViPO objective would strengthen the results.

For now, I will maintain my original score, although I am open to increasing it. I would prefer to discuss these remaining concerns with the AC and other reviewers before making a final decision.

Thank you for the thoughtful follow-up and your continued consideration.

W2. Reward hacking and high-gradient regions

Training reward vs. independent evaluation:

StepReward: GRPOReward: TVRLVBench: GRPOVBench: TVRL
0*−0.105−0.10554.0954.09
50−0.102−0.09854.3256.74
100−0.098−0.09454.5457.69
200−0.093−0.09553.3457.46
300−0.088−0.09451.5456.32

Training reward is the critic's mean log-probability reward on training prompts, averaged over steps [X−5, X+5]; step 0 is approximate (maximum reward: 0). VBench at step 100 is the paper's three-seed mean; other nonzero checkpoints are single-seed.

From step 100 to 300, GRPO's training reward rises (−0.098→−0.088), while its VBench score falls (54.54→51.54). This suggests reward over-optimization but does not prove reward hacking. TVRL's training reward is nearly flat (−0.094→−0.095), and VBench declines less (57.69→56.32). Changes are small since many checks already have high Yes likelihoods; TVRL also declines after step 100. This does not suggest that routing worsens over-optimization, nor prove it prevents it.

Are high-gradient regions semantic errors? Not necessarily, and our method does not assume that they are. The advantage decides the direction of the update. The map indicates which tokens the reward is most sensitive to. Thus, the map measures reward sensitivity, not error. The revised paper's Limitations clarify that these maps neither identify errors nor guarantee local corrections given shared DiT parameters.

Our evidence is that this sensitivity works as a routing signal:

  1. Prompt correspondence (R6): Cross-prompt maps remove the gain (54.55 vs. 54.54 uniform), while maps from another rollout of the same prompt and check retain most of it (56.72). The signal is largely prompt/check-dependent.
  2. Question-wise optimization (A5): With frame-level VLM-gradient credit, a single averaged ratio yields 54.96, whereas per-question ratios achieve 56.47. This supports question-wise routing.
  3. Question dependence (R5): On a fixed object mask in the same video, credit mass rises from 11.67% to 13.31% when the question concerns that object (lift 1.04×→1.14×; +1.65 pp, p=0.0034). The effect is modest but question-dependent.
  4. Independent outcomes: VBench-2.0 improves, and blind human raters prefer TVRL in 36.3% of comparisons versus 23.3% for GRPO (40.4% ties).

Measuring overlap with errors would require region-level annotations; we do not claim local error detection or correction.

W3. ViPO-style comparison

We agree that the combined baseline (54.85) is single-seed and not a validated reproduction of full ViPO. Our ViPO-style baseline was implemented from the published method description rather than the official codebase. We report its implementation differences and limit claims to our adaptation.

Implementation. We follow ViPO's Perceptual Structuring Module: DINOv2 patch features undergo PCA (K=3); the components are min-max normalized, inverted, and combined using explained-variance weights. We use DINOv2 ViT-L/14 on 20 sampled frames and align the map to our latent grid. We differ from ViPO in three respects: 3×3 aggregation instead of Gaussian smoothing (σ=1); frame-level rather than per-position clipped ratios; and group size 4 rather than 8. Backbone, reward, sampler, learning rate, clipping range, and step budget are matched within our pipeline. The combined row therefore evaluates ViPO-style perceptual credit with frame-local ratios, not ViPO's full per-position objective.

Controlled evidence. Our conclusion does not rely on the combined row alone. Three-seed experiments separately show: (i) DINOv2 maps obtain 55.48 versus 57.69 for reward-gradient maps under our weighted-ratio formulation; and (ii) on the same frame-level VLM-gradient maps, local frame ratios obtain 52.95 versus 56.47 with our per-question ratios. The combined adaptation reaches 54.85 in one seed; we do not interpret its small difference from DINOv2-only (55.48). We plan to provide two further seeds in the revision.

These results support reward-derived, question-conditioned routing over the tested perceptual-map and local-ratio alternatives in our pipeline. They do not establish superiority over the full ViPO implementation. Our claim is that the reward model's own input sensitivity provides a useful routing signal for video GRPO, not an independent local error signal or a new RL objective. Thank you again for your careful review and consideration.

Comments on the paper

2 comments

Dear Reviewers and AC,

We have uploaded a revised version of the paper. All changes are marked in orange-red.

New results (named as in our responses)

  • Table R1: longer training, 100 to 300 steps (Appendix A.7)
  • Table R2: ViPO-style components in our setting (Appendix A.4)
  • Table R3: shared versus per-check advantage (Appendix A.4)
  • Table R4: check-level outcomes on held-out prompts (Appendix A.9)
  • Table R5: question-swap test on generated videos (Appendix A.6)
  • Table R6: cross-prompt and group map swaps, added as rows to Table 3 in the main text
  • Table R8: compute with the Qwen3.5-9B critic, which replaces the previous overhead table (Appendix A.5)
  • Table R9: second backbone, Wan2.2-TI2V-5B (Appendix A.8)

Corrections and text changes

  • Table 1: the Overall of VideoScore2/TVRL (55.59) and UnifiedReward2/GRPO (55.02) now equals the mean of the five dimensions. The range of gains in the abstract and introduction is now 0.93 to 3.15.
  • The claim is restated as reward-sensitivity weighting of a shared advantage (abstract, introduction, Figure 2 caption, conclusion), and a Limitations paragraph is added.
  • Section 3.3 now discusses the shared advantage and the per-check alternative. The Table A3 row is renamed "Per-check advantage".
  • New single-seed results are marked with a dagger, and we draw no conclusions from small differences among them.

We thank the reviewers again for their detailed feedback and are willing to answer further questions.

Best,

Authors

Dear Reviewers, AC, SAC,

We would like to thank Reviewers 8o94, NbpE, and HUVW, the AC, and the SAC for the time and care they have put into this submission.

The paper is better because of this process. Your comments led us to state our claim more precisely, to correct an errors, and to add controls and analyses that the first version was missing. We see these as your contributions to the work, and we are appreciate for them.

We have posted replies to the latest follow-up comments. We hope you will have a chance to read them and have a discussion before the final decision, and we are happy to answer any further questions during the discussion.

Thank you again.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.