acceptodds
Under review as a conference paper at ICLR 2027

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

Abstract

Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.

Primary area:
generative models
Keywords:
Video Reward Model, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_AjfpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RewardVerse, a rubric-guided video reward modeling framework designed to mitigate "scalar drift"—the tendency of unconstrained multi-modal LLMs to produce collapsed or context-dependent scores. Instead of direct scoring, the method inserts a dynamic rubric generator that produces query-adaptive evaluation criteria (themes, weights, tips) without seeing the candidate video. A scorer then evaluates videos against these rubrics using a soft-logits readout to bypass natural language generation bias. The system is trained via Rubric-Guided Policy Optimization (RGPO), a two-stage GRPO-based algorithm that first warms up the scorer with offline seed rubrics and then jointly optimizes the generator and scorer using only 30 preference pairs per dimension. Experiments on the 16-dimensional EvalVerse benchmark and VGRB show state-of-the-art pointwise and pairwise performance, alongside downstream improvements in video generation.

Critical strengths
  1. The identification of scalar drift—specifically score-range collapse and high variance under context shift—as a core failure mode for video reward models is well-motivated. The initial 2x2 diagnostic study cleanly isolates the effects of rubric injection versus soft-logits decoding, providing solid empirical grounding for the subsequent architectural choices.
  2. Decoupling rubric generation from the candidate video is an elegant design choice. By conditioning the generator solely on the query, the framework structurally prevents the model from reverse-engineering lenient or strict criteria based on the video's visual quality, safeguarding evaluation objectivity while maintaining query adaptivity.
  3. The two-stage RGPO training pipeline is architecturally sound. Warming up the scorer with fixed seed rubrics before introducing dynamic generation avoids the highly under-determined joint optimization problem.
  4. Achieving competitive results with extreme data efficiency highlights the practical value of the rubric bottleneck as an inductive bias for low-data reward modeling.
Critical weaknesses
  1. The pairwise flip rate remains critically high (56.2%) even after rubric injection (Appendix A.1.6). While used to justify abandoning pairwise evaluation, such severe residual position bias calls into question the fundamental reliability of pointwise soft-logits extracted from the same underlying MLLM.
  2. Baseline comparisons rely on heuristic dimension mapping (Table 12). Since external baselines do not natively support all 16 EvalVerse dimensions, manually mapping their coarse outputs likely disadvantages them, making it unclear whether reported gains stem from the rubric mechanism or simply native alignment with the test taxonomy.
  3. Downstream RL validation is extremely narrow. Evaluating GRPO fine-tuning solely on Visual Quality (Section 5.4) fails to demonstrate whether dynamic rubrics generalize to optimizing complex temporal or narrative aspects like Logic or Pacing across the claimed 16 dimensions.
  4. Seed rubric synthesis relies entirely on Gemini-3.1-Pro. The paper lacks sensitivity analysis on how Stage 1 warm-up performs when the quality or diversity of these teacher-generated synthetic rubrics varies, leaving the student's performance ceiling poorly understood.
Questions for authors
  1. A detailed analysis of how the manual dimension mapping (Table 12) impacts baseline performance in Table 1, specifically whether evaluating baselines only on their native dimensions alters the overall ranking.
  2. Ablation results isolating the contribution of the soft-logits readout from the dynamic rubric during Stage 2 training, verifying whether continuous expectation extraction alone accounts for a significant portion of the PLCC/SRCC gains over natural language baselines.
  3. Downstream GRPO fine-tuning results on at least one cognitive or temporal dimension to validate that the query-adaptive rubrics effectively guide optimization beyond low-level visual quality.
  4. Sensitivity analysis regarding the seed rubric pool size and quality, clarifying how many Propose-Verify-Revise iterations or what minimum number of seed rubrics are required before Stage 1 scorer warm-up degrades.
  5. Clarification on the inference-time computational overhead introduced by the dynamic rubric generator, given that the scorer must now process structured JSON prompts containing multiple themes and tips for every video evaluated.
2 comments

Thank you for calling the candidate-independent rubric design “an elegant design choice” and recognizing its data efficiency. We hope our response will resolve your concerns.

W1 — Pairwise position bias versus the deployed pointwise scorer.

The 56.2% pair-swap flip rate (Table 8) measures two-video A/B prompting, where swapping video positions can change the judgment. RewardVerse instead scores each video separately under the same query-only rubric. For this deployed pointwise scorer, Table 10 shows prompt sensitivity falling 0.206→0.085 after RGPO.

W2 & Q1 — Baseline mapping.

Because some Table 1 baselines reuse coarse scores across fine-grained dimensions, we also test on VGRB's VQ and TA tasks. RewardVerse ranks first among the cross-dataset baselines on both (Table 2). On EvalVerse, giving each baseline's native scoring heads equal weight changes PLCC/SRCC by at most 0.03/0.03, while RewardVerse remains first.

W3 & Q3 — Logic (D15) downstream RL.

We added Logic-guided downstream GRPO (full results in our response to ShwE, W1 & Q1). Compared with the VideoReward-rewarded arm, RewardVerse achieves higher StoryEval event completion (0.9524 vs. 0.8988), DreamRunner transition scores (0.7467 vs. 0.7223), and the mean of four EvalVerse-based Logic sub-aspects (4.183 vs. 3.817), extending downstream validation to temporal and narrative quality.

W4 & Q4 — Seed-rubric sensitivity.

Seed quality: With themes and weights fixed, reducing tips from four to one per theme lowers Stage-1 PLCC/SRCC from 0.456/0.364 to 0.442/0.328. PVR convergence: Propose–verify–revise converges within four rounds, with 97.9% of accepted candidates requiring at most one revision. Seed quantity: From a candidate pool capped at 200 per dimension, MCR² shortlists five representative rubrics; we only select one highest-quality rubric per dimension for Stage-1 training. We have clarified this in paper.

Q2 — Soft-logits vs. dynamic rubrics.

Continuous expectation extraction alone does not explain the Stage-2 gains. The rubric-free scorer trained with soft-logits reaches 0.439/0.351 (Table 3e). With soft-logits unchanged, continuing scorer training under fixed rubrics reaches 0.467/0.380, while learning dynamic rubrics under the same Stage-2 budget reaches 0.530/0.416:

Soft-logits configurationPLCC ↑SRCC ↑
Rubric-free, trained (Table 3e)0.4390.351
Fixed rubric + continued scorer (matched Stage 2)0.4670.380
Learned dynamic rubric (Full RGPO)0.5300.416

With soft-logits held fixed, the +0.063 PLCC / +0.036 SRCC gain over the matched control supports the benefit of learning dynamic rubrics.

Q5 — Measured rubric inference overhead.

Table 15 reports 1.7 s/video for rubric-free soft-logits and 2.0 s/video for rubric-guided soft-logits (+0.3 s, approximately 18%). The generator produces all five themes and tips in one JSON rubric per query; the scorer then obtains all five theme scores in a single completion per video, not five separate forward passes. Thus, for one candidate, the pipeline needs one rubric-generation call and one scoring call. The rubric is reused across candidates sharing a query, amortizing the generation cost as Tgen/NT_{gen}/N.

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3086

Reviewer_ShwEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Video reward models tend to produce scores that bunch up or shift with the prompt, making the reward unreliable. This paper proposes RewardVerse, which inserts a rubric into the scoring process. The model generates weighted evaluation themes from the prompt, scores the video on each, and combines them into a final score. The model is trained in two stages with only 30 labeled pairs per aspect, and evaluated on preference prediction and as a reward for video generation.

Critical strengths
  1. The method is data-efficient. With only 480 preference pairs, Table 9 shows the bias dropping from +0.610 to +0.164 and PLCC rising from 0.240 to 0.505.
  2. Controlled ablations isolate which design choice fixes which symptom rather than reporting one aggregate gain, and makes the scalar-drift narrative credible.
Critical weaknesses
  1. The claim that RewardVerse provides a robust RL reward far outruns the evidence. The downstream experiment optimizes a single dimension with a single model, and the VBench quality moves only from 0.808 to 0.809, which cannot support a general claim about serving as a reliable RL reward.
  2. The second problem of scalar drift is bypassed rather than resolved. Even with the rubric, the pairwise flip rate stays at 56.2%, and the method sidesteps this by switching to a pointwise protocol instead of fixing it.
  3. The per-theme scores behind the interpretability claim are also never validated against humans.
Questions for authors
  1. The downstream RL result uses one dimension and one generation model with a VBench-Quality change of 0.808 to 0.809. Can the RL reward claim be supported on additional dimensions or models, and is the single-dimension gain outside noise?
  2. RGPO is trained only on Qwen2.5-VL-7B. Does the two-stage procedure yield comparable gains on a different backbone, or is the effect specific to this model?
  3. Only the aggregate margin is calibrated. How reliable are the individual per-theme scores, given that the interpretability claim depends on them?
  4. The pairwise flip rate remains 56.2% after adding the rubric. Is there an analogous diagnostic showing that the deployed pointwise scores are stable under context shift, rather than only less affected than the pairwise protocol?
4 comments

Thank you for recognizing the method's data efficiency and the credibility of our controlled scalar-drift ablations. We hope our response will resolve your concerns.

W1 & Q1 — Downstream RL.

In the original Wan2.2-A14B VQ experiment, Imaging Quality—the directly optimized quality metric—improves 0.640→0.653 (Table 13). Our VBench-Quality (6D) aggregate includes both image-quality and consistency measures; optimizing only VQ leaves a small decrease in Subject Consistency (0.898→0.894), moderating the aggregate change (0.808→0.809), whereas the VideoReward-rewarded arm declines to 0.804.

We additionally optimized Logic (D15) and evaluated against a VideoReward-rewarded arm:

EvaluationMetric / sub-aspectRewardVerseVideoReward
StoryEvalEvent completion (LLaVA-OV-7B)0.95240.8988
StoryEvalEvent completion (Qwen2.5-VL-7B)0.71630.6996
DreamRunnerTransition0.74670.7223
EvalVerse-based Logic (1–5)Multi-shot rationality4.0003.800
EvalVerse-based Logic (1–5)Narrative continuity4.2003.800
EvalVerse-based Logic (1–5)Scene consistency4.3333.867
EvalVerse-based Logic (1–5)Spatial continuity4.2003.800
EvalVerse-based Logic (1–5)Mean4.1833.817

We also ran VQ-guided RL on Wan2.1-T2V-1.3B, obtaining 0.727 vs. 0.530 Imaging Quality for the RewardVerse- and VideoReward-rewarded arms, respectively.

W2 & Q4 — Pointwise stability and pairwise position bias.

Table 10 directly tests the deployed pointwise scorer under three paraphrased instructions per video: normalized prompt sensitivity drops 0.206→0.085 after RGPO. The 56.2% flip rate in Table 8 measures direct two-video A/B prompting, where changing the input order can change an MLLM's judgment. This is a known issue in pairwise MLLM judging, which we do not claim to solve. RewardVerse instead scores each video independently under the same query-only rubric, without an A/B order variable.

W3 & Q3 — Theme-level reliability.

We compared individual per-theme scores with human ratings on 100 sampled cases and obtained 91.2% rating-level agreement, directly evaluating the reliability of theme-level judgments.

Q2 — Full two-stage RGPO on InternVL3.5-4B.

We trained both RGPO stages on InternVL3.5-4B, using a different backbone:

InternVL3.5-4BPLCC ↑SRCC ↑
Base0.3850.331
Stage 10.4210.358
Stage 2 (RGPO)0.4670.397

Across the same 16 EvalVerse dimensions, the full two-stage training improves PLCC 0.385→0.467 and SRCC 0.331→0.397.

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3086

Thank you for the detailed rebuttal. The additional RL experiments and cross-backbone results substantially address my concerns about generalizability. The clarification on pointwise stability and the new theme-level human evaluation are also helpful.

While some limitations remain, I find the rebuttal convincing overall and have increased my rating from 2 to 3.

Thank you for your time and thoughtful review! We sincerely appreciate your positive assessment and consideration of our work!

Reviewer_T6N1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RewardVerse, a rubric-guided video reward modeling framework designed to address scalar drift in reward scores. The paper shows that existing reward models often produce compressed and inconsistent scores across different videos and evaluation contexts. RewardVerse tackles this problem by first generating query-adaptive rubrics consisting of evaluation themes, weights, and scoring guidelines, which are then used to evaluate videos through a soft-logits scoring mechanism. The authors also propose Rubric-Guided Policy Optimization (RGPO), a two-stage training approach that first warms up the scorer using self-evolving seed rubrics and then optimizes the rubric generator through GRPO while continuing to calibrate the scorer using human preference margins. The method is trained using only 480 preference pairs across 16 evaluation dimensions. RewardVerse achieves strong pointwise correlations with human ratings on EvalVerse and competitive pairwise accuracy on VideoGen-RewardBench. The authors also demonstrate its effectiveness in downstream video generation, where it improves visual quality while largely preserving other generation capabilities.

Critical strengths
  1. The proposed method provides a more structured and interpretable way of evaluating videos by separating rubric generation from scoring. The rubrics are generated based on the query without seeing the candidate video, which helps reduce potential bias in scoring while allowing the evaluation criteria to adapt to different prompts.

  2. The paper identifies an important problem in existing video reward models, where the reward scores are often compressed or inconsistent across different evaluation contexts. The authors provide empirical evidence of this issue and show that rubric-guided scoring can help mitigate scalar drift.

  3. The paper provides strong experimental results on both pointwise and pairwise evaluation benchmarks, outperforming several existing video reward models. The authors also conduct ablation studies to demonstrate the effectiveness of different components of their proposed framework.

Critical weaknesses
  1. The paper provides ablations for different components of RGPO, including a static seed-rubric variant (Table 3(b)), but it is still unclear how much of the improvement comes from dynamically generating rubrics rather than improving the scorer through additional training. Ablation (b) removes Stage 2 entirely, so the full model receives roughly twice the optimization steps and continued scorer calibration. The authors should compare against a static-rubric baseline trained with the same optimization objectives and training budget. This would help establish whether the dynamic rubric generation is actually necessary.

  2. The paper claims that RewardVerse is highly data-efficient, requiring only 480 preference pairs. However, the seed rubric generation process relies on Gemini-3.1-Pro to generate, verify, and revise rubrics using these preference pairs. It is not clear how much additional computation and supervision this process introduces. The authors should report the cost of generating these rubrics and discuss whether similar improvements could be achieved by directly using the teacher model to supervise the scorer.

  3. As shown in Table 12, several baselines do not directly support all the evaluation dimensions, and the authors use the same reward scores for multiple dimensions. Since RewardVerse is specifically trained to evaluate these dimensions, this could give it an advantage. The authors should discuss this limitation and provide a more controlled comparison to better establish the improvements over existing methods.

Questions for authors

See weaknesses

4 comments

Thank you for recognizing the “strong experimental results” and the structured, interpretable rubric design. We hope our response will resolve your concerns.

W1 — Fixed-rubric control with matched training budget.

Following your suggestion, we continued Stage-2 scorer training with fixed rubrics, matching the Full RGPO steps and calibration settings:

Stage-2 settingPLCC ↑SRCC ↑
Fixed seed rubric + continued scorer (same budget)0.4670.380
Learned dynamic rubric — Full RGPO0.5300.416

Learning dynamic rubrics adds +0.063 PLCC / +0.036 SRCC beyond the continued-scorer control.

W2 — Teacher cost.

480 pairs (30 per dimension) are the entire human-label budget. Gemini-3.1-Pro uses those pairs to generate, verify and revise seed rubrics once offline (approximately 0.72M input / 0.33M output tokens; $5.40 API cost). RGPO training and deployment make no teacher calls. Direct teacher-score supervision might improve scorer performance, but would also transfer the teacher's own score-scale calibration bias rather than explicitly align rewards with human rating margins.

W3 — Baseline mapping.

To avoid relying only on the coarse-to-fine mappings in Table 12, we also evaluate on VGRB's VQ and TA tasks. RewardVerse achieves the best results among the cross-dataset baselines on both (Table 2). We also re-average Table 1 with equal weight for each baseline's native scoring head. PLCC/SRCC change by at most 0.03/0.03, and RewardVerse remains first.

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3086

I thank the authors for their detailed clarifications, which have addressed most of my concerns. I therefore maintain my initial positive score of 3 and remain inclined to recommend acceptance of the paper.

Thank you for your time and thoughtful review! We sincerely appreciate your positive assessment and consideration of our work!

Reviewer_HTXPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RewardVerse addresses pointwise video reward modeling through query-specific rubrics generated without inspecting the candidate video. A shared model produces themes, weights, and tips, then aggregates per-theme soft-logit scores. Two-stage RGPO warms up the scorer with teacher-generated seed rubrics, then learns rubric generation alongside human-margin calibration. The paper trains with 480 preference pairs and evaluates EvalVerse correlations across 16 dimensions, external pairwise agreement on VGRB, component ablations, and visual-quality fine-tuning of Wan-2.2-A14B.

Critical strengths

Query-only rubrics provide a useful evaluation interface. Candidates can share prompt-specific criteria rather than having criteria generated after each video is inspected (§4.1, p. 5). Compared with VisionReward’s fixed checklist as described in Table 12 (p. 19), the substantive addition is learned, query-conditioned themes, weights, and tips. These criteria are inspectable and reusable, although inspectability alone does not establish explanation faithfulness.

The reported gains are meaningful. Full RGPO reaches macro PLCC/SRCC of 0.530/0.416, versus 0.439/0.351 for rubric-free training with the same 480-pair budget (Table 3, p. 9). On external VGRB visual quality, accuracy without ties reaches 0.660, versus the strongest non-oracle baseline’s 0.630 (Table 2). These results support practical usefulness. They are more informative about the method than Table 1’s off-the-shelf comparisons, where coarse native scores sometimes serve as proxies for fine-grained dimensions (Table 12).

Critical weaknesses

The benefit of learning the criteria is not isolated. Section 5.4 attributes the 0.074 PLCC gain over Table 3(b) to adaptive rubrics. However, removing Stage 2 also removes further scorer training; Stage 2 adds another epoch and changes the calibration weight from 0.2 to 0.3 (§A.4.2, p. 21). This establishes the benefit of the second stage, not specifically generator learning over continued calibration with fixed rubrics.

The stabilization account goes beyond what the diagnostics establish. In the controlled Qwen comparison, adding rubrics to soft-logit scoring expands the score range, but bias rises from +0.546 to +0.771 and decided-pair accuracy falls from 0.795 to 0.754 (V2→V3, Table 4, p. 13). Normalized paraphrase sensitivity is comparable rather than improved: 0.179 versus 0.199 (Table 7, p. 16). Tables 9–10 do show better correlation, mean bias, and normalized stability after RGPO; that supports the trained pipeline, not a general reliability benefit from rubric injection alone. The claim that compressed rewards imply near-flat downstream gradients (§3.1) also needs qualification when advantages are group-normalized.

Questions for authors

Novelty clarification: What substantive differences in objectives or updates distinguish RGPO from Rubric-ARM and ARR-RPO? A direct comparison would materially affect my contribution assessment.

Targeted evidence request: Starting from Stage 1, how does continued scorer calibration with fixed rubrics perform under the Stage-2 update budget and calibration settings? This is the most useful control for separating learned criterion adaptation from additional scorer training.

Split clarification: What is the size of the validation set used to select checkpoint 45? Please specify its overlap rules, by video and prompt, with training, the 994-sample test set, and the diagnostic pools (§A.2; §A.4.2). The current description does not establish leakage, but leaves evaluation independence unclear.

VGRB clarification: How are tie thresholds selected, and what paired uncertainty accompanies the 0.6-percentage-point text-alignment gain with ties? Also, §5.3 should distinguish its comparison against VisionReward from the strongest baseline: for visual quality without ties, VideoScore2 scores 0.630, making the gain over the best non-oracle baseline 3.0 points, not 7.0.

3 comments

Thank you for noting “the reported gains are meaningful” and that query-only rubrics provide a useful evaluation interface. We hope our response will resolve your concerns.

W1 & Q2 — Equal-budget control for Stage-2 gains.

We added the requested control, using the same Stage-1 initialization, Stage-2 update budget and calibration coefficient, but keeping the rubric fixed:

Stage-2 settingPLCC ↑SRCC ↑
Fixed seed rubric + continued scorer (same budget)0.4670.380
Learned dynamic rubric — Full RGPO0.5300.416

Full RGPO exceeds the fixed-rubric control by +0.063 PLCC / +0.036 SRCC, supporting the benefit of learned criteria beyond continued scorer training.

W2 — Scalar drift.

In Table 4, untrained rubric injection expands score resolution but does not yet correct the backbone's scoring bias; human alignment and relative stability improve after RGPO training (Tables 9–10). We have clarified this distinction in the paper and refined the claim in §3.1: score compression weakens reward resolution rather than necessarily producing near-flat gradients under group-normalized GRPO.

Q1 — Related methods.

Rubric-ARM targets text-response preference judging and alternates RL updates to its rubric generator and judge for pairwise decisions. RewardVerse targets continuous pointwise video reward modeling: RGPO uses GRPO for rubric generation, but trains the scorer with human score-margin calibration, not judge-side policy gradients (Eqs. 9–12). ARR-RPO targets image generation and editing: it builds rubrics offline, uses a frozen VLM judge, and updates the downstream generation policy rather than training a reward model.

Q3 — Validation and split.

The validation set contains 100 videos, with no video/prompt overlap with training or the 994-video test set. Stage-2 checkpoint 45 is selected using validation macro-PLCC, not test scores. Our identifier- and prompt-level audit confirms no training–test overlap. The 240-video diagnostic pool and 48-video prompt-perturbation pool are sampled from the held-out test set and are used only for analysis, not training or checkpoint selection.

Q4 — VGRB ties and baseline.

For TA with ties, the 0.6-percentage-point gain (0.471 vs. 0.465) corresponds to 132 additional correct decisions across 22,079 evaluated pairs. We follow VGRB's official Ties Matter protocol: predict a tie when ∣sA−sB∣≤τ|s_A-s_B|\leq\tau, sweep candidate thresholds τ\tau, and report the highest three-class accuracy with ties, using the same procedure for every model. Thank you for catching the VQ comparison: VideoScore2 scores 0.630 without ties, so RewardVerse's 0.660 leads by 3.0 rather than 7.0 percentage points; we have corrected this.

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3086

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3086

Reviewer_SkrFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RewardVerse is designed to evaluate a generated video using its generation prompt and a target evaluation dimension. A rubric generator produces five weighted criteria without seeing the candidate video and a scorer evaluates the video against each criterion by using continuous scores derived from rating token probabilities.

It introduces Rubric-Guided Policy Optimization. It first trains the scorer using seed rubrics constructed through a teacher-driven proposal, verification and revision process. Next, it optimizes the rubric generator using GRPO while supervising the scorer to match its preference margins to human score differences. The paper's central claim is that explicit rubrics, continuous scoring and RGPO mitigate scalar drift, which encompasses score range collapse and context dependent changes.

Critical strengths
  1. The paper addresses the problem of scalar drift in scores generated by MLLMs. It provides useful diagnostics and analysis of the phenomenon. These include score ties, scoring bias, prompt perturbations, and training ablations.

  2. The evaluation considers explicit dimensions of video quality, which is potentially useful for both diagnosis and optimization.

3 The results show improvements in overall PLCC and SRCC performances over the reported baselines. Table 9 shows PLCC increasing along with a decrease in mean bias after RGPO under the same dynamic rubric protocol.

Critical weaknesses
  1. VideoReward is designated as "oracle" in Table 2, but the justification for excluding it from the competitive comparison is unclear. The VideoReward paper describes its training data and the benchmark dataset separately. It does not demonstrate training-test overlap. Since VideoReward outperforms RewardVerse in Table 2, the state-of-the-art claim depends on this exclusion. The authors should provide evidence supporting the "oracle" designation or consider VideoReward as a competitive baseline.

  2. The calibration objective in Equation 7 leaves query dependent offsets unconstrained. For example, one query can produce scores of 4.5 and 4.0 while another produces 2.5 and 2.0 for candidates with the same respective human ratings. Both pairs preserve a human margin of 0.5, so the loss cannot distinguish them. The authors need to evaluate the cross-query offsets, as a reduction in pooled bias (Table 9) does not establish cross-query calibration. The rubric provides criteria and tips but does not explicitly define what each rating level means.

  3. The ablations don't clearly isolate the effects of learning rubrics. In Table 3, removing Stage 2 also removes the additional scorer calibration and optimization steps, while removing Stage 1 changes the initialization and thus the trajectory of optimization. Removing tips doesn't isolate the contributions of learned weights, structured context or themes. The training objective and update budget of the rubric-free trained baseline also need clarification.

Questions for authors

Apart from the questions in the Weaknesses section. I have the following additional questions:

  1. The authors mention "Verified rubrics are pooled into Vd, deduplicated by a two-level Jaccard filter". How are semantically similar rubrics deduplicated ? For example, how are “Watch for feet sliding on the ground” and “Check that planted feet stay in place” recognized as semantically equivalent despite different wording ?

  2. How does the RGPO training fare on different backbones beyond the Qwen-vl model family ?

  3. Table 4 shows that when moving from V2 to V3, the standard deviation increases but bias also worsens and decide-pair accuracy decreases from 0.795 to 0.754. Almost all pairs are already decided by V2. Can the authors explain this tradeoff and its implications for the claimed benefit of score decompression?

5 comments

Thank you for recognizing our “useful diagnostics and analysis” of scalar drift. We hope our response will resolve your concerns.

W1 — VideoReward.

We evaluate VGRB as an out-of-distribution (OOD) benchmark. RewardVerse trains on only 480 EvalVerse preference pairs, without VGRB-specific data, and achieves the best VQ and TA results among models compared under the same OOD setting (Table 2). VideoReward, by contrast, was trained on 182K preference pairs covering the same VQ, MQ, and TA axes as VGRB, giving it a natural advantage on this benchmark. Q-Save, which shares co-authors with VideoReward, also treats VideoReward separately in its cross-dataset VGRB comparison. We report VideoReward's full results and have revised paper to label it a benchmark-related reference, rather than an “oracle”.

W2 — Margin calibration.

While margin calibration does not explicitly constrain query-dependent offsets, these offsets cancel in within-query preference margins and GRPO advantages. Reward modeling commonly uses pairwise preference learning as well as pointwise score regression. For RL, a reward model mainly needs to rank candidate videos for the same query and preserve meaningful reward gaps, rather than reproduce absolute human ratings (Ouyang et al., 2022). VideoReward similarly finds preference learning more effective than direct score regression, especially with limited data. With only 30 human preference pairs per dimension, adding an absolute-score MSE loss reduces PLCC/SRCC from 0.530/0.416 to 0.518/0.379, supporting our choice of margin-based calibration in this low-data setting.

We also evaluated cross-query offsets. The median absolute query-level mean scoring error decreases 0.86→0.75 points after RGPO, while the share of total squared error explained by between-query mean differences decreases 81%→62%. Rubrics define what to evaluate; human-margin training calibrates the resulting relative reward, rather than prescribing a fixed verbal definition for each score level.

W3 — Ablations.

We added a matched-budget Stage-2 control and ablations of rubric weights and tips:

SettingPLCC ↑SRCC ↑
Stage 1, 4 tips/theme (Table 3b)0.4560.364
Stage 1, 1 tip/theme0.4420.328
Fixed rubric + continued scorer (matched Stage 2)0.4670.380
Full RGPO, uniform weights0.5140.408
Full RGPO, learned weights0.5300.416

Under the same Stage-1 initialization and Stage-2 scorer update budget, learned rubrics outperform fixed rubrics by +0.063 PLCC / +0.036 SRCC, beyond continued scorer calibration. The uniform-weight and reduced-tip controls further assess weights and tips; Table 3(a) evaluates Stage-1 initialization (0.431/0.355), and Table 3(c) evaluates hierarchical tips (0.497/0.398). The rubric-free V2-Training baseline (Table 3e) optimizes only the margin calibration loss LcalL_{\mathrm{cal}} (Eq. 7) using rubric-free soft-logits scores, with the same 480 pairs and total update budget (0.439/0.351).

Q1 — Deduplication.

Jaccard is a lexical filter, not a synonym detector: it removes near-verbatim duplicates from stop-word-filtered rubric blocks (thresholds 0.65 within an EvalVerse path and 0.85 across paths). In the reviewer's example, “Watch for feet sliding on the ground” and “Check that planted feet stay in place” have token Jaccard 0.125, so their wording alone does not trigger lexical deduplication. Both may remain as tips under the same ground-contact theme. MCR² subsequently selects diverse representative rubrics in embedding space, reducing semantic redundancy at the rubric-selection stage.

Q2 — Backbone transfer.

We trained both RGPO stages on InternVL3.5-4B and evaluated 994 videos across the same 16 EvalVerse dimensions:

InternVL3.5-4BPLCC ↑SRCC ↑
Base0.3850.331
Stage 10.4210.358
Stage 2 (RGPO)0.4670.397

Both stages improve performance on this second backbone, with +0.082 PLCC / +0.066 SRCC over the base model.

Q3 — Decompression vs. calibration.

Table 4 evaluates untrained protocols. V2 already decides 239/240 pairs; rubric injection is meant to improve continuous score resolution, not the nearly saturated decided-pair count. Adding rubrics raises score std 0.258→0.471 (+82.6%), but without training, the wider range does not automatically correct backbone bias: bias changes +0.546→+0.771 and pair accuracy 0.795→0.754.

This is why RGPO calibration is necessary. Under the same dynamic-rubric protocol, training improves PLCC 0.240→0.505, reduces bias +0.610→+0.164 (Table 9), and lowers normalized prompt sensitivity 0.206→0.085 (Table 10). Rubrics increase score resolution; RGPO aligns these scores with human preferences.

Thank you for the detailed response and additional experiments. The matched-budget fixed-rubric control and clarification of the rubric-free baseline does address my concerns. In particular, the improvement over continued scorer training with fixed rubrics provides stronger evidence that the learned rubric pipeline contributes beyond additional scorer updates. The explanation of lexical deduplication and subsequent embedding-based selection also addresses my question. Regarding margin calibration, I agree that query-dependent offsets cancel in within-query comparisons and group-normalized advantages. The additional MSE experiment supports the chosen objective in the tested low-data setting, and the query-level error measurements are helpful. My remaining concerns are as follows:-

  1. Replacing “oracle” with “benchmark-related reference” (in W1) is an improvement. However, sharing VQ/MQ/TA axes and using more training data do not alone establish that a benchmark is in-distribution. Could you specify the criterion used to classify models into the OOD comparison group and apply it consistently across baselines? Please also qualify the headline performance claim to reflect this comparison scope and the stronger VideoReward results.

  2. Could you define a “query” in the new analysis (in W2) and report the number of unique queries and videos per query? Please provide the formula for the 81%→62% statistic and the corresponding absolute between-query error contribution or overall MSE, so that the change can be interpreted independently of its denominator.

  3. Please clarify whether the fixed rubric control matches the scorer loss coefficients, update schedule, and checkpoint selection procedure in full RGPO. Were uniform weights used throughout training or substituted only at evaluation? If available, uncertainty estimates for these comparisons would help assess the smaller contribution of learned weights.

Finally, I appreciate the clarification that Table 4 concerns untrained protocols. I suggest making this distinction explicit in the revised claims

Thank you for your quick reply. We are glad that most of your concerns have been addressed and hope the following clarifications resolve the remaining questions.

R1 — VGRB comparison.

To evaluate out-of-distribution (OOD) generalization, we test RewardVerse on VGRB, where it achieves the best VQ and TA results among all baselines except VideoReward in Table 2, using only 480 human preference pairs.

We define OOD/cross-dataset baselines as models developed independently of VGRB without VGRB-specific training or tuning. VideoReward was developed alongside VGRB and trained on 182K preference pairs covering VQ/MQ/TA, so we report it separately as a benchmark-related reference, without assuming training–test overlap. We acknowledge its stronger VGRB results (VQ/TA: 0.756/0.722 vs. 0.660/0.623). To provide an additional comparison, we evaluate both models on GenAI-Bench (VideoScore-Bench), which uses overall human preferences rather than VQ/MQ/TA-specific labels. Using both models' VQ scores under the same per-aspect evaluation protocol:

ModelAcc w/ TieAcc w/o Tie
RewardVerse (VQ)0.4280.655
VideoReward (VQ)0.3650.554

RewardVerse achieves higher accuracy in this VQ-based comparison, despite using 379× fewer human preference pairs. We will qualify the headline claims to reflect the comparison scope and VideoReward's stronger VGRB results.

R2 — Cross-query calibration.

Following §4.1, a query is q=(d,p)q=(d,p), consisting of evaluation dimension dd and generation prompt pp. Filtering the held-out EvalVerse test set for queries with at least two videos yields 62 unique queries and 183 videos (2.95 videos/query). This subset is distinct from the 240-video set in Table 9.

For residual eq,i=s^q,i−sq,ie_{q,i}=\hat{s}_{q,i}-s_{q,i}, query mean eˉq\bar e_q, and global mean eˉ\bar e, we compute:

\mathrm{MSE}=\frac{1}{N}\sum_{q,i}e_{q,i}^2.$$ Here, $B$ measures between-query variation in mean residuals, and $B/\mathrm{MSE}$ its share of total squared error. | Metric | Pre-RGPO | Post-RGPO | |---|---:|---:| | Overall MSE | 0.765 | 0.790 | | Between-query contribution ($B$) | 0.622 | **0.490** | | $B/\mathrm{MSE}$ | 81% | **62%** | | Median absolute query-mean error | 0.86 | **0.75** | **Between-query variation decreases by 21.2%**, and median absolute query-mean error by 12.8%. Although overall MSE slightly increases, these results indicate reduced between-query offsets rather than improved overall MSE. **R3 — Ablation settings.** The fixed-rubric control matches Full RGPO in **Stage-1 initialization, Stage-2 scorer loss coefficients, update schedule, and training steps**. Both use the same **100-video validation set** and select checkpoints by validation macro-PLCC: Step 45 for Full RGPO and the final checkpoint for the fixed-rubric control. Uniform weights ($w_k=0.2$) are substituted **only at inference**, without retraining. PLCC/SRCC changes from **0.530/0.416 to 0.514/0.408**. We have also clarified in **§3.2 and Table 4** that V1–V4 compare untrained protocols, as highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.