- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
A scalar reward on a whole video cannot say which requirement the video failed, so the paper builds a fine-grained rubric reward for RL post-training. Its core design is a two-rubric architecture: an original-prompt rubric decomposed offline by Gemini 3.7-Flash into binary items over six requirement categories and frozen before training independently of the sampled rewrites, plus a per-rewrite condition rubric compiled by a frozen Qwen3.8-27B from the condition actually fed to the video model with a coverage check; items are scored by a frozen VLM's yes/no logit difference over a soft temporal window, zeroed unless prerequisites pass, and turned into item-wise group-relative advantages driving one on-policy GRPO step on the rewriter and one coefficients-preserving-sampling step on the video policy.
It also contributes Video-Rubrics-60K and trains three backbones (LingBot-Video 30B-A3B, Cosmos3-Nano, MiniMax-H3) for 1000 iterations. Evaluation is on RBench only: Overall 0.614 → 0.625, 0.584 → 0.591, 0.660 → 0.669, plus ablations over the update strategy and the window policy.
Critical strengths
-
Freezing the original-prompt rubric independently of the sampled rewrites is the paper's best idea, and it answers a failure mode that quietly ruins most rubric-based RL.
-
The serializer/compiler split and its fixed checkpoints; the item schema with source span, grounding flag, evidence scope and prerequisites.
Critical weaknesses
-
The measured effect is smaller than the unreported noise, and the per-category pattern contradicts the paper's own motivation. RBench Overall moves by +0.011, +0.007 and +0.009. The paper reports no seeds, no standard deviations, no error bars, no confidence intervals and no significance tests anywhere.
-
The central architectural claim is never ablated, the verifier is never validated, and the Base rows appear to be copied rather than re-run. Three gaps compound: 1) No anchoring ablation. There is no experiment comparing the frozen original-prompt rubric against one compiled only from the rewrite. The two ablations that exist — update strategy (Table 2) and window policy (Table 3) — are both orthogonal to the design decision identified above as the main contribution. So strength (1) is asserted, not demonstrated. 2) No verifier validation. The reward is a yes/no logit difference from a frozen Qwen3.8-27B. There is no agreement with human labels, no precision/recall on the binary items, no per-category scorer reliability, and no reward-hacking analysis — despite 1000 iterations of RL against exactly that scorer, and despite the rewriter being explicitly rewarded for coverage of items it does not itself produce. The hard 0.5 gating threshold adds an unmeasured discontinuity. 3) Base rows. App. E.1 states the nine Base category scores "are retained with their reported precision" from published entries. That sits uneasily with Sec. 3.1's claim that Base and Video-Rubrics "use the same evaluation inputs and generation settings", and with a +0.011 headline delta: if Base comes from a different setting or rounding, the delta is not like-for-like.
Questions for authors
How accurate is the frozen item scorer? A human-labeled subset (even 300–500 items) with per-category precision/recall, or item-level agreement with the RBench evaluator, would let me judge whether the reward measures what it claims. Also: how sensitive are results to the hard 0.5 gating threshold?