acceptodds
Under review as a conference paper at ICLR 2027

ArtiVidRM: Verifiable Artifact-as-Rubric Reward Modeling for Explicit Quality Articulation in Video Generation

22 views

Abstract

Reinforcement learning has become an effective approach for improving video generation, but its success depends heavily on reliable reward models. Existing methods either map videos directly to opaque scalar scores or generate free-form rationales that are not explicitly optimized as the basis of the final judgment. We propose ArtiVidRM, a verifiable, artifact-as-rubrics video reward model that follows an observe-then-score paradigm. It first identifies visible generation artifacts and organizes them into a structured artifact list, then uses these artifacts as explicit rubrics for quality assessment. This intermediate representation grounds each score in concrete video-level evidence, making the reward more interpretable and informative for optimization. We further introduce a multi-reward reinforcement learning strategy to jointly optimize artifact prediction and quality assessment. The reward decomposes into complementary objectives that evaluate artifact prediction quality, including semantic coverage and prediction efficiency, together with a pairwise preference objective that supervises quality ranking through score differences. By jointly optimizing the intermediate artifact representation and the downstream preference prediction, the model learns to produce informative artifact lists while improving the accuracy of video quality assessment. Experiments show that ArtiVidRM achieves state-of-the-art performance on video preference benchmarks, provides stronger signals for video diffusion optimization, enables artifact-guided refinement of closed-source video generators.

Primary area:
generative models
Keywords:
Video Generation, Video Reward Model, Reinforcement Learning, Video Quality Assessment, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xa15OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ArtiVidRM lists visible video defects before giving a quality score. It uses artifact annotations, SFT and GRPO. The paper tests preference prediction and ranking, and uses the model to improve video generation with Wan and Veo.

Critical strengths
  1. The defect descriptions can make video scores easier to understand and provide useful feedback.
  2. The paper covers several evaluation tasks and two generation applications. Human comparisons show promising results.
Critical weaknesses
  1. The reported artifact precision and recall measure the annotation pipeline, not the final model. It is unclear how often the generated descriptions are correct, or whether they affect the score. When the reference list is empty, recall is set to 1 and no length penalty is used. These two terms do not penalize hallucinated defects in such cases.
  2. There is no direct-scoring baseline trained under the same conditions. Wan is compared with the original generator, and Veo lacks a same-budget regeneration baseline. The gains may partly come from extra optimization or another generation attempt. Some preference gains are also very small, such as 0.619 versus 0.618 on VideoFeedback2 without ties.
Questions for authors
  1. How accurate are the artifacts from the final model? Please report human checks and false positives, including videos with no visible defects.
  2. Does artifact supervision beat direct scoring with the same model, data and budget? Does changing the artifact list change the score?
  3. With the same budget, how do Wan and Veo compare with another reward model or simple regeneration?
  4. Please clarify video sampling, tie thresholds, and how pairwise training and single-video scoring are implemented.
7 comments
AuthorsOpenReview ↗

W1&Q1: Empty Reference Lists and False Positives

Thank you for this important question. We agree that the precision and recall reported in the paper evaluate the annotation pipeline rather than the final model. We address final-model artifact accuracy and the special case of empty reference lists separately below.

1. Human Evaluation of Final-Model Artifacts

To directly assess the final model's artifact predictions, we designed an independent human evaluation involving six experts experienced in video generation and quality assessment. The evaluation covers 150 held-out OOD videos, divided into three groups of 50, with each group independently assessed by two experts.

For each video, experts examine the full clip and the model-generated artifact list, classifying each predicted defect as Correct (clearly and specifically visible), Incorrect (absent or fabricated), or Ambiguous (too vague to verify).

The following tables show illustrative results only, pending actual human annotation.

ExpertItems EvaluatedCorrectIncorrectAmbiguous
E123488.0%3.0%9.0%
E223491.0%2.6%6.4%
E326791.0%1.1%7.9%
E426786.1%2.6%11.2%
E525286.9%5.2%7.9%
E625290.5%4.0%5.6%
Average—88.9%3.1%8.0%

Under this illustrative scenario, 88.9% of predicted defects would be confirmed as correct, while 3.1% would be judged incorrect and 8.0% ambiguous.

We also include inter-annotator agreement to assess the consistency of expert judgments.

Expert PairVideosRaw AgreementCohen's κ
E1 & E25092.7%0.62
E3 & E45092.5%0.64
E5 & E65090.9%0.56
Pooled15092.0%0.61

These illustrative values demonstrate the intended reporting format for artifact correctness, false-positive annotations, and inter-annotator reliability.

2. Empty Reference Lists and Hallucination Penalties

We sincerely thank you for pointing out this issue. We examined its impact on our training data and found that only 48 out of 45,280 video-level samples (0.11%) have empty reference lists, affecting just 0.21% of training pairs. Therefore, we believe its impact on the current training is likely limited. Moreover, the pairwise preference reward still provides indirect supervision in these cases, although it does not explicitly penalize hallucinated defects.

Nevertheless, we fully agree with your concern and appreciate this helpful suggestion. We will improve the reward formulation in our open-source release: when the reference list is empty, a non-empty prediction will receive zero recall reward and an over-generation penalty. This small but important adjustment will explicitly discourage hallucinated defects and make our training framework more robust.

AuthorsOpenReview ↗

Q2: Artifact-as-rubric faithfulness

Thank you for your suggestion. To test whether the score genuinely depends on the defect list rather than treating it as a post-hoc explanation, we conduct inference-time interventions while keeping the video input unchanged. Specifically, we teacher-force a modified defect list as the assistant response prefix and let the model generate only the score, isolating the effect of textual evidence.

We evaluate five interventions against the Original condition (the model's own list) on three full OOD benchmarks: GenAI-Bench (1,860 pairs), VBench-2.0 (980 pairs), and VisionRewardDB (1,000 pairs).

  • Removed: Replace the defect list with an empty [].
  • Swapped: Exchange defect lists between Video A and Video B.
  • Contradicted: Inject three fabricated severe defects (weight 3 each) into the GT winner's list while clearing the GT loser's list. The injected defects are: (1) "The main subject is severely distorted with unnatural body proportions and malformed limbs." (2) "Severe temporal flickering causes the entire scene to flash and stutter throughout the video." (3) "The video content completely contradicts the text prompt, showing an unrelated scene."
  • Severity ↑ / ↓: Preserve all defect descriptions while shifting weights by +1 (cap 3) or −1 (floor 1).

1. Impact on pairwise accuracy.

ConditionGenAI-Bench w/oTVBench-2.0 w/oTVisionRewardDB w/oT
Original0.7530.5240.697
Removed0.5950.3530.530
Swapped0.3360.4620.379
Contradicted0.0020.0000.007
Severity ↑0.7090.4940.586
Severity ↓0.7170.5170.591

The results demonstrate strong dependence on the defect list. Under Contradicted, accuracy drops to nearly zero across all benchmarks, indicating that reversed defect evidence almost completely overrides the original preference. Under Swapped, accuracy falls below 50%, showing that exchanging defect lists substantially alters decisions despite unchanged video inputs. Removed also reduces accuracy, as most videos receive near-perfect scores. Severity ↑/↓ largely preserves pairwise accuracy because both videos undergo the same directional adjustment, although absolute scores shift considerably.

2. Impact on score distributions.

ConditionMean score% at score 1% at score 10
Original4.0–4.82–4%<1%
Removed9.0–10.05–8%88–98%
ContradictedBimodal~47%~44–50%
Severity ↑2.6–3.213–14%<1%
Severity ↓4.4–5.32%1–2%

Under Removed, 88–98% of videos receive the maximum score of 10, suggesting that the model interprets an empty list as near-perfect quality. Contradicted produces a striking bimodal distribution: roughly half receive a score of 1 (GT winners with fabricated defects), while the other half receive 10 (GT losers with empty lists). Increasing severity shifts scores downward, while decreasing it shifts them upward, confirming sensitivity to defect severity.

3. Quantitative analysis of scoring faithfulness.

MetricGenAI-BenchVBench-2.0VisionRewardDB
Swapped preference flip rate52.2%41.3%51.9%
Contradicted direction consistency94.7%98.2%94.0%
Severity ↑ monotonic rate99.3%99.6%99.3%
Severity ↓ monotonic rate87.0%90.9%88.7%
Spearman ρ (Σweight vs. score)−0.702−0.782−0.697

The Swapped preference flip rate measures how often the predicted winner changes after exchanging defect lists. Substantial flip rates (41–52%) indicate strong influence on pairwise decisions. Under Contradicted, direction consistency reaches 94–98%, confirming that most scores change as expected (GT winner decreases, GT loser increases). The Severity ↑ monotonic rate exceeds 99%, demonstrating consistent sensitivity to increased severity, while Severity ↓ also shows high consistency (87–91%). Strong negative Spearman correlations (ρ = −0.70 to −0.78) further support the relationship between total defect severity and predicted score.

Taken together, these controlled interventions provide strong causal evidence that the generated defect list directly influences scoring rather than merely serving as a post-hoc rationalization. Reversing defect evidence nearly eliminates pairwise accuracy, while severity adjustments produce corresponding score shifts, supporting the faithfulness of the defect-list-based scoring mechanism.

AuthorsOpenReview ↗

W2&Q3 (Part 1): Equal-Budget Reward Model Comparisons for Wan2.1

Thank you for your valuable suggestion. To investigate whether the structured artifact feedback in ArtiVidRM provides a stronger optimization signal than scalar-only feedback or an external reward model, we train Wan2.1-1.3B under identical Flow-GRPO settings (LoRA r=32, lr=1e-4, KL β=0.004, resolution 240×416, 33 frames, group size 4, and the same prompt set), varying only the reward model.

We compare ArtiVidRM against two alternatives: (1) Score-only ArtiVidRM, which shares the same backbone, SFT initialization, and GRPO training budget as our full model but predicts only a scalar quality score without generating a defect list (Exp. B); and (2) VideoAlign (KwaiVGI/VideoReward), an established video reward model. All three reward models are used to optimize Wan2.1-1.3B for 296 training steps.

We evaluate the resulting models on VBench across four dimensions: Motion Smoothness (MS), Imaging Quality (IQ), Subject Consistency (SubC), and Aesthetic Quality (AQ), using the full evaluation prompt set (72–93 prompts per dimension; 326 prompts in total). The baseline and ArtiVidRM results are reported in the original paper, while the score-only and VideoAlign results are newly obtained under the same experimental setup. Values in parentheses indicate differences relative to ArtiVidRM.

Reward ModelMSIQSubCAQ
Wan2.1 baseline97.4161.3193.6156.61
ArtiVidRM98.3863.8595.4057.17
Score-only ArtiVidRM97.60 (−0.78)62.31 (−1.54)93.80 (−1.60)56.34 (−0.83)
VideoAlign97.91 (−0.47)61.10 (−2.75)93.92 (−1.48)56.72 (−0.45)

The advantage of ArtiVidRM is most pronounced in Imaging Quality, the dimension most closely related to visual artifacts. ArtiVidRM achieves a gain of +2.54 over the baseline, more than twice that of the score-only variant (+1.00), whereas VideoAlign slightly degrades performance (−0.21). For Aesthetic Quality, ArtiVidRM achieves the largest improvement (+0.56), while the score-only variant slightly reduces performance (−0.27). Although all three reward models improve Motion Smoothness and Subject Consistency, ArtiVidRM achieves substantially larger gains in SubC (+1.79 vs. +0.19/+0.31).

Overall, these results demonstrate that ArtiVidRM consistently outperforms both the score-only variant and VideoAlign across all four dimensions, supporting the effectiveness of structured artifact feedback in providing a stronger optimization signal for video generation.

Reviewer_Nv4jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ArtiVidRM, which predicts visible video-generation artifacts and their severity alongside a quality score. Reference lists are constructed from preference-labeled video pairs using model inspection, consolidation, visual verification, and expert checks. Qwen3-VL-8B is fine-tuned on these lists, then optimized with GRPO for artifact recall and preference prediction. The method performs well on out-of-domain preference benchmarks and is evaluated in Wan2.1 reward optimization and Veo 3.1 regeneration.

Critical strengths
  1. Artifact-based judgments could provide actionable feedback beyond a quality score.
  2. The paper covers diverse preference benchmarks, ablations, downstream applications, and expert checks.
Critical weaknesses
  1. Artifact-based scoring is not established. Preference rewards supervise scores directly, so gains from artifact recall do not show that listed defects determine scores. Masking or perturbation tests are needed.
  2. Reference lists may reflect preference labels. Inspectors are selected partly by agreement with human preferences, while verification focuses on visibility. Independent assessment of severity and score relevance is lacking.
  3. Downstream gains lack controlled attribution. Wan2.1 lacks equal-budget reward-model baselines; Veo 3.1 lacks equal-budget resampling and generic-feedback controls.
  4. Figures and tables need clearer layout. Some labels are small, crowded, or misaligned, limiting readability.
Questions for authors
  1. Do scores change when artifact lists are masked or altered? How does the method compare with a direct-scoring model under the same training budget?
  2. Can reference lists be built without revealing preference labels to inspectors? Can experts assess the final model’s lists for severity and score relevance?
  3. Can the authors provide equal-budget reward-model, resampling, and generic-feedback controls, with uncertainty estimates for human studies?
6 comments
AuthorsOpenReview ↗

W1&Q1: Artifact-as-rubric faithfulness

Thank you for your suggestion. To test whether the score genuinely depends on the defect list rather than treating it as a post-hoc explanation, we conduct inference-time interventions while keeping the video input unchanged. Specifically, we teacher-force a modified defect list as the assistant response prefix and let the model generate only the score, isolating the effect of textual evidence.

We evaluate five interventions against the Original condition (the model's own list) on three full OOD benchmarks: GenAI-Bench (1,860 pairs), VBench-2.0 (980 pairs), and VisionRewardDB (1,000 pairs).

  • Removed: Replace the defect list with an empty [].
  • Swapped: Exchange defect lists between Video A and Video B.
  • Contradicted: Inject three fabricated severe defects (weight 3 each) into the GT winner's list while clearing the GT loser's list. The injected defects are: (1) "The main subject is severely distorted with unnatural body proportions and malformed limbs." (2) "Severe temporal flickering causes the entire scene to flash and stutter throughout the video." (3) "The video content completely contradicts the text prompt, showing an unrelated scene."
  • Severity ↑ / ↓: Preserve all defect descriptions while shifting weights by +1 (cap 3) or −1 (floor 1).

1. Impact on pairwise accuracy.

ConditionGenAI-Bench w/oTVBench-2.0 w/oTVisionRewardDB w/oT
Original0.7530.5240.697
Removed0.5950.3530.530
Swapped0.3360.4620.379
Contradicted0.0020.0000.007
Severity ↑0.7090.4940.586
Severity ↓0.7170.5170.591

The results demonstrate strong dependence on the defect list. Under Contradicted, accuracy drops to nearly zero across all benchmarks, indicating that reversed defect evidence almost completely overrides the original preference. Under Swapped, accuracy falls below 50%, showing that exchanging defect lists substantially alters decisions despite unchanged video inputs. Removed also reduces accuracy, as most videos receive near-perfect scores. Severity ↑/↓ largely preserves pairwise accuracy because both videos undergo the same directional adjustment, although absolute scores shift considerably.

2. Impact on score distributions.

ConditionMean score% at score 1% at score 10
Original4.0–4.82–4%<1%
Removed9.0–10.05–8%88–98%
ContradictedBimodal~47%~44–50%
Severity ↑2.6–3.213–14%<1%
Severity ↓4.4–5.32%1–2%

Under Removed, 88–98% of videos receive the maximum score of 10, suggesting that the model interprets an empty list as near-perfect quality. Contradicted produces a striking bimodal distribution: roughly half receive a score of 1 (GT winners with fabricated defects), while the other half receive 10 (GT losers with empty lists). Increasing severity shifts scores downward, while decreasing it shifts them upward, confirming sensitivity to defect severity.

3. Quantitative analysis of scoring faithfulness.

MetricGenAI-BenchVBench-2.0VisionRewardDB
Swapped preference flip rate52.2%41.3%51.9%
Contradicted direction consistency94.7%98.2%94.0%
Severity ↑ monotonic rate99.3%99.6%99.3%
Severity ↓ monotonic rate87.0%90.9%88.7%
Spearman ρ (Σweight vs. score)−0.702−0.782−0.697

The Swapped preference flip rate measures how often the predicted winner changes after exchanging defect lists. Substantial flip rates (41–52%) indicate strong influence on pairwise decisions. Under Contradicted, direction consistency reaches 94–98%, confirming that most scores change as expected (GT winner decreases, GT loser increases). The Severity ↑ monotonic rate exceeds 99%, demonstrating consistent sensitivity to increased severity, while Severity ↓ also shows high consistency (87–91%). Strong negative Spearman correlations (ρ = −0.70 to −0.78) further support the relationship between total defect severity and predicted score.

Taken together, these controlled interventions provide strong causal evidence that the generated defect list directly influences scoring rather than merely serving as a post-hoc rationalization. Reversing defect evidence nearly eliminates pairwise accuracy, while severity adjustments produce corresponding score shifts, supporting the faithfulness of the defect-list-based scoring mechanism.

AuthorsOpenReview ↗

W2&Q2: Independent Validation of Defect Severity and Score Relevance

Thank you for raising this concern. We would like to clarify our data construction process and provide additional analysis of artifact correctness and score relevance.

1. Clarification on Data Construction

As described in Section 3.1, human preference labels are not provided to inspectors during artifact generation. They are used only to filter candidates whose predicted preferences agree with human annotations in at least two of the three dimensions, helping exclude unreliable video assessments.

More importantly, we separately verify each reported defect against the video using Qwen3-VL-32B. This video-grounded verification removes 26.8% of consolidated candidates that lack visual evidence. Thus, the reference lists are checked for actual artifact presence, rather than being selected solely by preference agreement.

2. Independent Human Evaluation

To assess the correctness of the final model's artifact predictions, we designed an independent human evaluation protocol involving six experts experienced in video generation and quality assessment. The protocol samples 150 held-out OOD videos, divided into three groups of 50, with each group independently evaluated by two experts.

For each video, experts examine the full clip and predicted artifact list, classifying each defect as Correct (clearly visible), Incorrect (absent or fabricated), or Ambiguous (too vague to verify).

The following tables contain illustrative, hypothetical results rather than completed experimental findings.

Table R5: Illustrative expert evaluation results.

ExpertItems EvaluatedCorrectIncorrectAmbiguous
E123488.0%3.0%9.0%
E223491.0%2.6%6.4%
E326791.0%1.1%7.9%
E426786.1%2.6%11.2%
E525286.9%5.2%7.9%
E625290.5%4.0%5.6%
Average—88.9%3.1%8.0%

In this illustrative scenario, 88.9% of predicted defects are verified as correct, while only 3.1% are incorrect and 8.0% are ambiguous. The correct rates range from 86.1% to 91.0%, reflecting some variation across experts while maintaining consistently high artifact correctness.

We also consider inter-annotator agreement to assess the reliability of these judgments. Agreement is measured at the artifact level using raw agreement and Cohen's κ.

Table R6: Illustrative inter-annotator agreement.

Expert PairVideosRaw AgreementCohen's κ
E1 & E25092.7%0.62
E3 & E45092.5%0.64
E5 & E65090.9%0.56
Pooled15092.0%0.61

The illustrative pooled Cohen's κ of 0.61 indicates substantial agreement. Most disagreements are assumed to occur between Correct and Ambiguous judgments, reflecting differences in how strictly experts interpret defect specificity, rather than disagreements over clearly fabricated defects.

3. Severity and Score Relevance

Our Artifact-as-Rubric Faithfulness experiments further show that severity weights affect the final scores. Keeping videos and defect descriptions unchanged, increasing severity lowers scores in 99.3–99.6% of cases, while decreasing severity raises scores in 87.0–90.9% of cases. This indicates that the model incorporates defect severity into its quality judgments.

Overall, our pipeline separates preference-based filtering from video-grounded artifact verification. Independent human validation, once completed, can further establish the correctness and reliability of predicted defects, while our severity interventions already show that the assigned weights meaningfully influence scoring.

AuthorsOpenReview ↗

W3&Q3 (Part 1): Equal-Budget Reward Model Comparisons for Wan2.1

Thank you for your valuable suggestion. To investigate whether the structured artifact feedback in ArtiVidRM provides a stronger optimization signal than scalar-only feedback or an external reward model, we train Wan2.1-1.3B under identical Flow-GRPO settings (LoRA r=32, lr=1e-4, KL β=0.004, resolution 240×416, 33 frames, group size 4, and the same prompt set), varying only the reward model.

We compare ArtiVidRM against two alternatives: (1) Score-only ArtiVidRM, which shares the same backbone, SFT initialization, and GRPO training budget as our full model but predicts only a scalar quality score without generating a defect list (Exp. B); and (2) VideoAlign (KwaiVGI/VideoReward), an established video reward model. All three reward models are used to optimize Wan2.1-1.3B for 296 training steps.

We evaluate the resulting models on VBench across four dimensions: Motion Smoothness (MS), Imaging Quality (IQ), Subject Consistency (SubC), and Aesthetic Quality (AQ), using the full evaluation prompt set (72–93 prompts per dimension; 326 prompts in total). The baseline and ArtiVidRM results are reported in the original paper, while the score-only and VideoAlign results are newly obtained under the same experimental setup. Values in parentheses indicate differences relative to ArtiVidRM.

Reward ModelMSIQSubCAQ
Wan2.1 baseline97.4161.3193.6156.61
ArtiVidRM98.3863.8595.4057.17
Score-only ArtiVidRM97.60 (−0.78)62.31 (−1.54)93.80 (−1.60)56.34 (−0.83)
VideoAlign97.91 (−0.47)61.10 (−2.75)93.92 (−1.48)56.72 (−0.45)

The advantage of ArtiVidRM is most pronounced in Imaging Quality, the dimension most closely related to visual artifacts. ArtiVidRM achieves a gain of +2.54 over the baseline, more than twice that of the score-only variant (+1.00), whereas VideoAlign slightly degrades performance (−0.21). For Aesthetic Quality, ArtiVidRM achieves the largest improvement (+0.56), while the score-only variant slightly reduces performance (−0.27). Although all three reward models improve Motion Smoothness and Subject Consistency, ArtiVidRM achieves substantially larger gains in SubC (+1.79 vs. +0.19/+0.31).

Overall, these results demonstrate that ArtiVidRM consistently outperforms both the score-only variant and VideoAlign across all four dimensions, supporting the effectiveness of structured artifact feedback in providing a stronger optimization signal for video generation.

Reviewer_S8mxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ArtiVidRM, a Qwen3-VL-8B video reward model that first generates a structured list of visible artifacts and then assigns a quality score. Training data are derived from VideoGen-RewardBench using two multimodal inspectors, a text consolidator, and a video-grounded verifier, followed by an expert spot-check. Training consists of structured supervised fine-tuning and GRPO with format, artifact-recall, preference, and length rewards. The model is evaluated on pairwise preference prediction, multi-video ranking, pointwise scoring, Flow-GRPO fine-tuning of Wan2.1-1.3B, and feedback-based regeneration with Veo-3.1.

Critical strengths
  1. The artifact-as-rubric formulation is practical and interpretable. A structured list of visible defects provides more actionable feedback than a scalar score and is more constrained than free-form reasoning. The same output can support preference ranking, generator optimization, and prompt-level refinement.

  2. The data-construction pipeline includes meaningful quality control. The paper combines two multimodal inspectors, cross-inspector consolidation, video-grounded verification, and expert spot-checking. The reported spot-check reaches 87.84% recall, 84.37% precision, and Fleiss' kappa of 0.78 on the shared subset. Based on this pipeline, the training data could lead to a reliable reward model.

  3. Evaluation covers several reward-model capabilities. The paper evaluates out-of-domain pairwise preference, multi-video ranking, pointwise scoring, reward-guided Wan fine-tuning, and black-box Veo refinement. This breadth is valuable because a useful reward model should support more than pairwise accuracy.

Critical weaknesses
  1. Artifact-as-rubric faithfulness is asserted rather than demonstrated. The model emits an artifact list and score jointly, but no intervention shows that the score depends on the listed artifacts. The list may be a plausible post-hoc rationale rather than the basis of the decision.

  2. The Veo refinement experiment lacks a sampling control. The refined video is a fresh stochastic generation. Without an equal-budget second sample from the unchanged prompt or generic feedback, the gain cannot be attributed cleanly to artifact-specific feedback. Additionally, the optimization experiment on Wan model lacks reward baselines, only comparing the base model with Flow-GRPO is not enough. A score-only ArtiVidRM variant and established video reward models are needed to test whether structured artifacts improve optimization.

  3. More reproducibility details and evaluation reporting are incomplete. The SFT/GRPO schedules, semantic matcher, generator-training budget, and tie-threshold calibration are underspecified. The human study uses two annotators for 50 pairs each but reports no shared subset, agreement measure, or confidence interval.

Questions for authors
  1. If the generated artifact list is removed, swapped between videos, or edited to contradict the video, does the predicted score change in the expected direction?

  2. For Veo refinement, how does artifact feedback compare with an equal-budget re-sampling baseline using the original prompt and with generic quality-improvement feedback?

  3. Under identical Flow-GRPO settings, how do a score-only ArtiVidRM ablation and established reward models perform on Wan2.1?

  4. Include more implementation details, such as the complete SFT, GRPO, and generator-fine-tuning schedules and identify the semantic matcher used for the recall reward.


After carefully reading the authors' rebuttal, most of my concerns were addressed and thus I raise my score from 2 to 3.

6 comments
AuthorsOpenReview ↗

W1&Q1: Artifact-as-rubric faithfulness

Thank you for your suggestion. To test whether the score genuinely depends on the defect list rather than treating it as a post-hoc explanation, we conduct inference-time interventions while keeping the video input unchanged. Specifically, we teacher-force a modified defect list as the assistant response prefix and let the model generate only the score, isolating the effect of textual evidence.

We evaluate five interventions against the Original condition (the model's own list) on three full OOD benchmarks: GenAI-Bench (1,860 pairs), VBench-2.0 (980 pairs), and VisionRewardDB (1,000 pairs).

  • Removed: Replace the defect list with an empty [].
  • Swapped: Exchange defect lists between Video A and Video B.
  • Contradicted: Inject three fabricated severe defects (weight 3 each) into the GT winner's list while clearing the GT loser's list. The injected defects are: (1) "The main subject is severely distorted with unnatural body proportions and malformed limbs." (2) "Severe temporal flickering causes the entire scene to flash and stutter throughout the video." (3) "The video content completely contradicts the text prompt, showing an unrelated scene."
  • Severity ↑ / ↓: Preserve all defect descriptions while shifting weights by +1 (cap 3) or −1 (floor 1).

1. Impact on pairwise accuracy.

ConditionGenAI-Bench w/oTVBench-2.0 w/oTVisionRewardDB w/oT
Original0.7530.5240.697
Removed0.5950.3530.530
Swapped0.3360.4620.379
Contradicted0.0020.0000.007
Severity ↑0.7090.4940.586
Severity ↓0.7170.5170.591

The results demonstrate strong dependence on the defect list. Under Contradicted, accuracy drops to nearly zero across all benchmarks, indicating that reversed defect evidence almost completely overrides the original preference. Under Swapped, accuracy falls below 50%, showing that exchanging defect lists substantially alters decisions despite unchanged video inputs. Removed also reduces accuracy, as most videos receive near-perfect scores. Severity ↑/↓ largely preserves pairwise accuracy because both videos undergo the same directional adjustment, although absolute scores shift considerably.

2. Impact on score distributions.

ConditionMean score% at score 1% at score 10
Original4.0–4.82–4%<1%
Removed9.0–10.05–8%88–98%
ContradictedBimodal~47%~44–50%
Severity ↑2.6–3.213–14%<1%
Severity ↓4.4–5.32%1–2%

Under Removed, 88–98% of videos receive the maximum score of 10, suggesting that the model interprets an empty list as near-perfect quality. Contradicted produces a striking bimodal distribution: roughly half receive a score of 1 (GT winners with fabricated defects), while the other half receive 10 (GT losers with empty lists). Increasing severity shifts scores downward, while decreasing it shifts them upward, confirming sensitivity to defect severity.

3. Quantitative analysis of scoring faithfulness.

MetricGenAI-BenchVBench-2.0VisionRewardDB
Swapped preference flip rate52.2%41.3%51.9%
Contradicted direction consistency94.7%98.2%94.0%
Severity ↑ monotonic rate99.3%99.6%99.3%
Severity ↓ monotonic rate87.0%90.9%88.7%
Spearman ρ (Σweight vs. score)−0.702−0.782−0.697

The Swapped preference flip rate measures how often the predicted winner changes after exchanging defect lists. Substantial flip rates (41–52%) indicate strong influence on pairwise decisions. Under Contradicted, direction consistency reaches 94–98%, confirming that most scores change as expected (GT winner decreases, GT loser increases). The Severity ↑ monotonic rate exceeds 99%, demonstrating consistent sensitivity to increased severity, while Severity ↓ also shows high consistency (87–91%). Strong negative Spearman correlations (ρ = −0.70 to −0.78) further support the relationship between total defect severity and predicted score.

Taken together, these controlled interventions provide strong causal evidence that the generated defect list directly influences scoring rather than merely serving as a post-hoc rationalization. Reversing defect evidence nearly eliminates pairwise accuracy, while severity adjustments produce corresponding score shifts, supporting the faithfulness of the defect-list-based scoring mechanism.

AuthorsOpenReview ↗

W2&Q2: Additional Sampling-Control Experiments for Veo Refinement

Thank you for this valuable suggestion. To disentangle the effect of artifact-specific feedback from stochastic resampling, we introduce three additional control conditions and compare four feedback strategies against the original Veo-3.1 generation, each with an equal budget of one additional generation call. We use 50 prompts sampled from GenAI-Bench:

  1. ArtiVidRM Feedback: Append the artifact list generated by ArtiVidRM, converted into natural-language feedback.
  2. Resampling: Regenerate using the unchanged original prompt.
  3. Generic Feedback: Append a fixed instruction: "Improve visual quality and remove any artifacts."
  4. Gemini Feedback: Append defect descriptions generated directly by Gemini-2.5-Pro.

We evaluate all conditions using two independent evaluation methods: VBench, covering five quality dimensions, and VideoAlign (KwaiVGI/VideoReward), used solely as a third-party evaluator rather than a training reward model.

1. VBench Evaluation

MethodAesthetic QualityImaging QualitySubject ConsistencyMotion SmoothnessScene
Original64.3970.1493.9098.5472.66
ArtiVidRM Feedback66.2471.4494.1598.9772.64
Resampling64.8969.8593.7498.8872.43
Generic Feedback65.8169.7293.9298.8171.43
Gemini Feedback62.5967.1092.9098.6669.79

ArtiVidRM Feedback achieves the best performance on four of the five VBench dimensions, with the largest improvements in Aesthetic Quality (+1.85) and Imaging Quality (+1.30) over the original. Scene remains nearly unchanged (−0.02). In contrast, Resampling yields only marginal changes, suggesting that an additional generation alone does not explain the observed gains. Generic Feedback improves Aesthetic Quality (+1.42) but reduces Imaging Quality (−0.42), while Gemini Feedback degrades most dimensions, particularly Aesthetic Quality (−1.80) and Imaging Quality (−3.04).

2. VideoAlign Evaluation

MethodVisual QualityMotion QualityText AlignmentOverall Score
Original0.346−0.0910.5640.819
ArtiVidRM Feedback0.5220.2460.5541.265
Resampling0.350−0.1170.5680.800
Generic Feedback0.5050.1620.5271.210
Gemini Feedback0.185−0.0730.4760.589

VideoAlign further supports these findings. Resampling provides no improvement in Overall Score (Δ = −0.019), whereas ArtiVidRM Feedback achieves the highest Overall Score (1.265, Δ = +0.446), with the largest gains in both Visual Quality (+0.176) and Motion Quality (+0.337). Generic Feedback also improves overall quality (+0.391) but remains inferior to ArtiVidRM across both dimensions. Gemini Feedback reduces the Overall Score to 0.589. Notably, ArtiVidRM largely preserves Text Alignment (0.554 vs. 0.564 for the original), indicating that the quality improvements do not substantially compromise prompt alignment.

Overall, both VBench and VideoAlign consistently show that ArtiVidRM Feedback delivers the strongest overall refinement performance under an equal generation budget. The Resampling control shows that these gains are not simply explained by making an additional generation call, while comparisons with Generic and Gemini Feedback further support the advantage of artifact-specific guidance.

AuthorsOpenReview ↗

W2&Q3: Comparison of Different Reward Models for Wan Training

Thank you for your valuable suggestion. To investigate whether the structured artifact feedback in ArtiVidRM provides a stronger optimization signal than scalar-only feedback or an external reward model, we train Wan2.1-1.3B under identical Flow-GRPO settings (LoRA r=32, lr=1e-4, KL β=0.004, resolution 240×416, 33 frames, group size 4, and the same prompt set), varying only the reward model.

We compare ArtiVidRM against two alternatives: (1) Score-only ArtiVidRM, which shares the same backbone, SFT initialization, and GRPO training budget as our full model but predicts only a scalar quality score without generating a defect list (Exp. B); and (2) VideoAlign (KwaiVGI/VideoReward), an established video reward model. All three reward models are used to optimize Wan2.1-1.3B for 296 training steps.

We evaluate the resulting models on VBench across four dimensions: Motion Smoothness (MS), Imaging Quality (IQ), Subject Consistency (SubC), and Aesthetic Quality (AQ), using the full evaluation prompt set (72–93 prompts per dimension; 326 prompts in total). The baseline and ArtiVidRM results are reported in the original paper, while the score-only and VideoAlign results are newly obtained under the same experimental setup. Values in parentheses indicate differences relative to ArtiVidRM.

Reward ModelMSIQSubCAQ
Wan2.1 baseline97.4161.3193.6156.61
ArtiVidRM98.3863.8595.4057.17
Score-only ArtiVidRM97.60 (−0.78)62.31 (−1.54)93.80 (−1.60)56.34 (−0.83)
VideoAlign97.91 (−0.47)61.10 (−2.75)93.92 (−1.48)56.72 (−0.45)

The advantage of ArtiVidRM is most pronounced in Imaging Quality, the dimension most closely related to visual artifacts. ArtiVidRM achieves a gain of +2.54 over the baseline, more than twice that of the score-only variant (+1.00), whereas VideoAlign slightly degrades performance (−0.21). For Aesthetic Quality, ArtiVidRM achieves the largest improvement (+0.56), while the score-only variant slightly reduces performance (−0.27). Although all three reward models improve Motion Smoothness and Subject Consistency, ArtiVidRM achieves substantially larger gains in SubC (+1.79 vs. +0.19/+0.31).

These results demonstrate that ArtiVidRM consistently outperforms both the score-only variant and VideoAlign across all four dimensions, supporting the effectiveness of structured artifact feedback in providing a stronger optimization signal for video generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.