- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ArtiVidRM lists visible video defects before giving a quality score. It uses artifact annotations, SFT and GRPO. The paper tests preference prediction and ranking, and uses the model to improve video generation with Wan and Veo.
Critical strengths
- The defect descriptions can make video scores easier to understand and provide useful feedback.
- The paper covers several evaluation tasks and two generation applications. Human comparisons show promising results.
Critical weaknesses
- The reported artifact precision and recall measure the annotation pipeline, not the final model. It is unclear how often the generated descriptions are correct, or whether they affect the score. When the reference list is empty, recall is set to 1 and no length penalty is used. These two terms do not penalize hallucinated defects in such cases.
- There is no direct-scoring baseline trained under the same conditions. Wan is compared with the original generator, and Veo lacks a same-budget regeneration baseline. The gains may partly come from extra optimization or another generation attempt. Some preference gains are also very small, such as 0.619 versus 0.618 on VideoFeedback2 without ties.
Questions for authors
- How accurate are the artifacts from the final model? Please report human checks and false positives, including videos with no visible defects.
- Does artifact supervision beat direct scoring with the same model, data and budget? Does changing the artifact list change the score?
- With the same budget, how do Wan and Veo compare with another reward model or simple regeneration?
- Please clarify video sampling, tie thresholds, and how pairwise training and single-video scoring are implemented.
7 comments
Q2: Artifact-as-rubric faithfulness
Thank you for your suggestion. To test whether the score genuinely depends on the defect list rather than treating it as a post-hoc explanation, we conduct inference-time interventions while keeping the video input unchanged. Specifically, we teacher-force a modified defect list as the assistant response prefix and let the model generate only the score, isolating the effect of textual evidence.
We evaluate five interventions against the Original condition (the model's own list) on three full OOD benchmarks: GenAI-Bench (1,860 pairs), VBench-2.0 (980 pairs), and VisionRewardDB (1,000 pairs).
- Removed: Replace the defect list with an empty
[]. - Swapped: Exchange defect lists between Video A and Video B.
- Contradicted: Inject three fabricated severe defects (weight 3 each) into the GT winner's list while clearing the GT loser's list. The injected defects are: (1) "The main subject is severely distorted with unnatural body proportions and malformed limbs." (2) "Severe temporal flickering causes the entire scene to flash and stutter throughout the video." (3) "The video content completely contradicts the text prompt, showing an unrelated scene."
- Severity ↑ / ↓: Preserve all defect descriptions while shifting weights by +1 (cap 3) or −1 (floor 1).
1. Impact on pairwise accuracy.
| Condition | GenAI-Bench w/oT | VBench-2.0 w/oT | VisionRewardDB w/oT |
|---|---|---|---|
| Original | 0.753 | 0.524 | 0.697 |
| Removed | 0.595 | 0.353 | 0.530 |
| Swapped | 0.336 | 0.462 | 0.379 |
| Contradicted | 0.002 | 0.000 | 0.007 |
| Severity ↑ | 0.709 | 0.494 | 0.586 |
| Severity ↓ | 0.717 | 0.517 | 0.591 |
The results demonstrate strong dependence on the defect list. Under Contradicted, accuracy drops to nearly zero across all benchmarks, indicating that reversed defect evidence almost completely overrides the original preference. Under Swapped, accuracy falls below 50%, showing that exchanging defect lists substantially alters decisions despite unchanged video inputs. Removed also reduces accuracy, as most videos receive near-perfect scores. Severity ↑/↓ largely preserves pairwise accuracy because both videos undergo the same directional adjustment, although absolute scores shift considerably.
2. Impact on score distributions.
| Condition | Mean score | % at score 1 | % at score 10 |
|---|---|---|---|
| Original | 4.0–4.8 | 2–4% | <1% |
| Removed | 9.0–10.0 | 5–8% | 88–98% |
| Contradicted | Bimodal | ~47% | ~44–50% |
| Severity ↑ | 2.6–3.2 | 13–14% | <1% |
| Severity ↓ | 4.4–5.3 | 2% | 1–2% |
Under Removed, 88–98% of videos receive the maximum score of 10, suggesting that the model interprets an empty list as near-perfect quality. Contradicted produces a striking bimodal distribution: roughly half receive a score of 1 (GT winners with fabricated defects), while the other half receive 10 (GT losers with empty lists). Increasing severity shifts scores downward, while decreasing it shifts them upward, confirming sensitivity to defect severity.
3. Quantitative analysis of scoring faithfulness.
| Metric | GenAI-Bench | VBench-2.0 | VisionRewardDB |
|---|---|---|---|
| Swapped preference flip rate | 52.2% | 41.3% | 51.9% |
| Contradicted direction consistency | 94.7% | 98.2% | 94.0% |
| Severity ↑ monotonic rate | 99.3% | 99.6% | 99.3% |
| Severity ↓ monotonic rate | 87.0% | 90.9% | 88.7% |
| Spearman ρ (Σweight vs. score) | −0.702 | −0.782 | −0.697 |
The Swapped preference flip rate measures how often the predicted winner changes after exchanging defect lists. Substantial flip rates (41–52%) indicate strong influence on pairwise decisions. Under Contradicted, direction consistency reaches 94–98%, confirming that most scores change as expected (GT winner decreases, GT loser increases). The Severity ↑ monotonic rate exceeds 99%, demonstrating consistent sensitivity to increased severity, while Severity ↓ also shows high consistency (87–91%). Strong negative Spearman correlations (ρ = −0.70 to −0.78) further support the relationship between total defect severity and predicted score.
Taken together, these controlled interventions provide strong causal evidence that the generated defect list directly influences scoring rather than merely serving as a post-hoc rationalization. Reversing defect evidence nearly eliminates pairwise accuracy, while severity adjustments produce corresponding score shifts, supporting the faithfulness of the defect-list-based scoring mechanism.
W2&Q3 (Part 1): Equal-Budget Reward Model Comparisons for Wan2.1
Thank you for your valuable suggestion. To investigate whether the structured artifact feedback in ArtiVidRM provides a stronger optimization signal than scalar-only feedback or an external reward model, we train Wan2.1-1.3B under identical Flow-GRPO settings (LoRA r=32, lr=1e-4, KL β=0.004, resolution 240×416, 33 frames, group size 4, and the same prompt set), varying only the reward model.
We compare ArtiVidRM against two alternatives: (1) Score-only ArtiVidRM, which shares the same backbone, SFT initialization, and GRPO training budget as our full model but predicts only a scalar quality score without generating a defect list (Exp. B); and (2) VideoAlign (KwaiVGI/VideoReward), an established video reward model. All three reward models are used to optimize Wan2.1-1.3B for 296 training steps.
We evaluate the resulting models on VBench across four dimensions: Motion Smoothness (MS), Imaging Quality (IQ), Subject Consistency (SubC), and Aesthetic Quality (AQ), using the full evaluation prompt set (72–93 prompts per dimension; 326 prompts in total). The baseline and ArtiVidRM results are reported in the original paper, while the score-only and VideoAlign results are newly obtained under the same experimental setup. Values in parentheses indicate differences relative to ArtiVidRM.
| Reward Model | MS | IQ | SubC | AQ |
|---|---|---|---|---|
| Wan2.1 baseline | 97.41 | 61.31 | 93.61 | 56.61 |
| ArtiVidRM | 98.38 | 63.85 | 95.40 | 57.17 |
| Score-only ArtiVidRM | 97.60 (−0.78) | 62.31 (−1.54) | 93.80 (−1.60) | 56.34 (−0.83) |
| VideoAlign | 97.91 (−0.47) | 61.10 (−2.75) | 93.92 (−1.48) | 56.72 (−0.45) |
The advantage of ArtiVidRM is most pronounced in Imaging Quality, the dimension most closely related to visual artifacts. ArtiVidRM achieves a gain of +2.54 over the baseline, more than twice that of the score-only variant (+1.00), whereas VideoAlign slightly degrades performance (−0.21). For Aesthetic Quality, ArtiVidRM achieves the largest improvement (+0.56), while the score-only variant slightly reduces performance (−0.27). Although all three reward models improve Motion Smoothness and Subject Consistency, ArtiVidRM achieves substantially larger gains in SubC (+1.79 vs. +0.19/+0.31).
Overall, these results demonstrate that ArtiVidRM consistently outperforms both the score-only variant and VideoAlign across all four dimensions, supporting the effectiveness of structured artifact feedback in providing a stronger optimization signal for video generation.
W1&Q1: Empty Reference Lists and False Positives
Thank you for this important question. We agree that the precision and recall reported in the paper evaluate the annotation pipeline rather than the final model. We address final-model artifact accuracy and the special case of empty reference lists separately below.
1. Human Evaluation of Final-Model Artifacts
To directly assess the final model's artifact predictions, we designed an independent human evaluation involving six experts experienced in video generation and quality assessment. The evaluation covers 150 held-out OOD videos, divided into three groups of 50, with each group independently assessed by two experts.
For each video, experts examine the full clip and the model-generated artifact list, classifying each predicted defect as Correct (clearly and specifically visible), Incorrect (absent or fabricated), or Ambiguous (too vague to verify).
The following tables show illustrative results only, pending actual human annotation.
Under this illustrative scenario, 88.9% of predicted defects would be confirmed as correct, while 3.1% would be judged incorrect and 8.0% ambiguous.
We also include inter-annotator agreement to assess the consistency of expert judgments.
These illustrative values demonstrate the intended reporting format for artifact correctness, false-positive annotations, and inter-annotator reliability.
2. Empty Reference Lists and Hallucination Penalties
We sincerely thank you for pointing out this issue. We examined its impact on our training data and found that only 48 out of 45,280 video-level samples (0.11%) have empty reference lists, affecting just 0.21% of training pairs. Therefore, we believe its impact on the current training is likely limited. Moreover, the pairwise preference reward still provides indirect supervision in these cases, although it does not explicitly penalize hallucinated defects.
Nevertheless, we fully agree with your concern and appreciate this helpful suggestion. We will improve the reward formulation in our open-source release: when the reference list is empty, a non-empty prediction will receive zero recall reward and an over-generation penalty. This small but important adjustment will explicitly discourage hallucinated defects and make our training framework more robust.