- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TVRL uses frozen reward-model input gradients to distribute a video-level GRPO advantage across spatial and temporal units. It improves VBench-2.0 over matched GRPO baselines across several samplers and reward models.
Critical strengths
-
Reward-derived routing addresses a relevant limitation of scalar-feedback video optimization.
-
Multi-seed experiments across three samplers and four reward models, together with blind human evaluation, support the empirical gains.
-
Ablations examine routing granularity, DINOv2-based maps, smoothing, and alternative ratio formulations.
Critical weaknesses
-
Spatial grounding remains weakly established. Appendix A.6 assigns only 52.7% of credit to the queried panel versus 50% under uniform routing. The shuffled control also changes the final map statistics, leaving semantic alignment incompletely isolated.
-
Shared advantages limit check-specific correction. A failed check can receive positive reinforcement when the overall rollout advantage is positive. The implications deserve stronger evaluation, particularly since controllability barely improves over matched GRPO.
-
Compute efficiency is unclear. The reported 18% overhead comes from imperfectly matched historical VideoAlign runs, rather than the default Qwen3.5-9B configuration. Comparisons at equal optimizer steps do not establish equal-compute benefits.
Questions for authors
-
Does routing remain beneficial against controls that preserve final-map smoothness and entropy while disrupting semantic correspondence?
-
Can you quantify outcomes for failed checks under positive shared advantages, and provide a controlled advantage ablation clarifying the historical variant in Table A3?
-
What are the default configuration’s GPU-hours, peak memory, and performance at matched wall-clock budgets?
4 comments
Q2. Failed checks under positive shared advantages; controlled ablation for Table A3
(1) Outcomes of failed checks. Among the checks that the base model fails on held-out prompts, TVRL fixes 45.5% vs. 25.5% for GRPO, and it breaks 5.5% of passing checks vs. 8.2% for GRPO (Table R4). A failed check lowers the reward of its rollout, so it receives a positive advantage only when the other checks of that rollout keep its reward above the group mean. Both methods train with the same shared advantage, so the rates above include the effect of all such updates; their net outcome is that failed checks are fixed more often, and more so with token routing. This is an aggregate measure: it does not isolate the individual rollouts in which a failed check received a positive advantage.
(2) Controlled ablation for Table A3. The "Token credit" variant uses frame-level credit with a separate advantage per check. Its controlled counterpart is the Frame row of Table 3 (same frame-level credit, shared advantage), which differs only in the advantage: 54.42 vs. 56.47 (Table R3). We will rewrite the caption to say so in the revised PDF.
Table R4. Failed checks on held-out prompts: 200 prompts × 3 seeds × 5 checks = 3,000 checks, of which 825 fail under the base model (p(Yes) < 0.5). Step-100 checkpoints, same seeds for all models. The scorer is the training critic, so we compare the two methods and do not interpret the absolute rates.
| Model | Check pass rate | Fix rate (base fail → pass) | Break rate (base pass → fail) |
|---|---|---|---|
| Base | 72.5% | – | – |
| GRPO | 73.6% | 25.5% | 8.2% |
| TVRL | 81.0% | 45.5% | 5.5% |
Table R3. Shared vs. per-check advantage. The second and third rows differ only in the advantage.
| Variant | Advantage | Credit granularity | Overall |
|---|---|---|---|
| GRPO (uniform) | shared | none | 54.54 |
| "Token credit" (Table A3) | per check | frame | 54.42 |
| Frame credit (Table 3) | shared | frame | 56.47 |
| TVRL | shared | 3×3 | 57.69 |
W3 and Q3. Compute for the default configuration; matched wall-clock
We agree that the VideoAlign pair does not measure the default configuration. Table R8 reports the default Qwen3.5-9B configuration from the logged training runs (64 A100 80GB GPUs, per-device batch size 2; mean time per update over updates 11–100). GRPO takes 710.5 s per update and TVRL 844.5 s (+18.9%), i.e. 1,263 vs. 1,501 GPU-hours per 100 updates, with an observed device-level peak memory of 54.6 vs. 77.9 GiB. The higher peak memory, close to the 80 GB capacity of the device, is a practical cost of the method.
The policy-optimization stage takes the same time in both runs (420 s). About 12 s of the difference falls in the rollout stage, where the critic's backward pass runs, and the rest comes from recomputing old-policy log-probabilities for weighted routing. Measured in isolation on one GPU, computing the credit maps for 5 questions raises the critic's time from 0.80 to 3.33 s per video and its peak memory from 18.1 to 21.0 GiB.
Matched wall-clock. In the time TVRL needs for 100 updates, GRPO can run 119. We did not evaluate GRPO at exactly 119 updates, so the following comparison is indicative, not exactly matched. Two results suggest that the extra updates would not close the gap. First, GRPO does not improve beyond 100 updates in our long-run experiment (54.54 at step 100, 53.34 at step 200, 51.54 at step 300; Table R1 in our response to Reviewer 8o94). Second, TVRL at step 50, which costs 59% of the wall-clock of GRPO's 100 updates, already reaches 56.74.
Table R8. Compute for the default Qwen3.5-9B configuration (SAGE, seed 42; 64 A100 80GB GPUs). Peak memory is the largest sampled device-level allocation on the reporting host.
| Method | s per update | Relative | GPU-hours per 100 updates | Peak GPU memory (observed) |
|---|---|---|---|---|
| GRPO | 710.5 | 1.00× | 1,263 | 54.6 GiB |
| TVRL | 844.5 | 1.19× | 1,501 | 77.9 GiB |
Thanks for your deliberate rebuttal, most of my concern have been resolved. I maintain my recommend for acceptance.
Thank you for your thoughtful feedback and for recognizing our efforts to address your concerns. Your suggestions helped us strengthen the experimental evidence and clarify the architectural contributions. We appreciate your time and constructive discussion.
We thank the reviewer for the positive assessment and for three concrete requests, which we answer with new measurements below.
On seeds. All new training runs in this response use a single seed because of the time and compute available during the rebuttal; results from the submission are means over three seeds. We will report three seeds for every new run in the final version. We treat the single-seed results as preliminary evidence and do not draw conclusions from small numerical differences. The tables labelled R are new and will be added to the revised PDF.
W1 and Q1. Spatial grounding; controls that preserve map smoothness and entropy
The A.6 test measures the question–position interaction (D = +5.39 pp, p = 0.0034, 10 of 12 groups), which is zero for any score-independent map. We agree that the absolute mass shift is modest in this constructed 2×4 setting. We add two cleaner tests.
Distribution-preserving control (Q1). We reuse real maps unchanged, so smoothness, entropy, and the weight distribution are identical by construction and only the correspondence to the video is broken (Table R6). With maps from a video of a different prompt, Overall falls from 57.69 to 54.55, the level of uniform GRPO (54.54). With maps from another rollout of the same prompt and check, Overall is 56.72. Routing therefore remains beneficial only when the maps match the prompt and its checks; much of the signal is conditioned on the prompt and the check rather than on the individual rollout.
Table R6. Map controls (SAGE, Qwen3.5-9B, 100 steps). The two swap rows are new and single-seed; the other rows are from the paper.
Question-swap test on generated videos. With the video and the object mask fixed, an object receives more credit when the question is about it than when it is about another object in the same video (Table R5). The effect is modest, in line with A.6: it supports question-conditioned sensitivity, not precise object-level localization.
Table R5. Question-swap test on generated videos with two segmentable objects (GroundingDINO + SAM masks). We ask "Does the video show a/an …?" once per object and measure the credit mass on a fixed object mask. Training critic configuration, 3×3 routing, coordinates as in Appendix A.6; lift = credit mass ÷ mask area, averaged over videos (1.0 = uniform). 12 videos pass a mask check fixed before any gradient was computed.
Difference: +1.65 pp (95% CI [+0.43, +3.27]; exact paired sign-flip test, p = 0.0034; positive in 10 of 12 videos). A.6 reports the same p-value because both tests have 12 units and, in both, 14 of the 4,096 sign assignments reach the observed mean. On all 20 attempted videos, without the mask check: +1.28 pp, positive in 18 of 20.
W2. Shared advantages limit check-specific correction
As stated in Sec. 3.3, this is a deliberate choice, and we now evaluate it. We tested the alternative of a separate advantage per check: with the same frame-level credit maps it yields 54.42 Overall, vs. 56.47 with the shared advantage and 54.54 for uniform GRPO (Table R3, in part 2 under Q2).
The shared advantage is the average of the per-check centered scores, so each check contributes to the shared reward and supplies its own routing map, while the sign of the update is shared across checks. The per-check variant instead gives each check its own update direction and magnitude. It also normalizes every check separately over only 4 rollouts, so checks on which the rollouts barely differ receive the same weight as checks on which they clearly differ. The shared advantage normalizes once, after pooling the 5 checks.
Controllability. We agree that it barely moves on HunyuanVideo-1.5 (31.57 → 31.64), where the gains concentrate in Creativity (+5.68) and Physics (+8.63). On a second backbone, Wan2.2-TI2V-5B, it improves from 21.63 (GRPO) to 26.58 (TVRL), with Overall 53.24 → 56.34 in a single-seed run, which suggests that the flat result is specific to that setting. We report the HunyuanVideo-1.5 result as it is and will note it in the limitations.