- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RAVEN-Eval, a rubric-guided LMM-as-a-judge framework for evaluating text-to-video and image-to-video generation models. It constructs 250 tasks and 4,669 generated videos, uses task-specific rubrics for order-reversed pairwise judgments with ties, and estimates model scores with a Davidson-style model. The paper also proposes anchor-based insertion of new models and reports leaderboards for 20 video generation models and 13 LMM judges. Validation includes comparisons with human rankings on 50 tasks, external leaderboards, rubric ablations, and simulated model insertion experiments.
Critical strengths
The paper addresses a practically important problem: evaluating video generation models is costly. It proposes using LMMs as judges and presents an end-to-end pipeline covering task construction, rubric-guided pairwise evaluation, and incremental model insertion. The two-level rubric provides criteria for both task-specific core requirements and general video quality, making the basis for judgments more explicit. The evaluation framework covers 20 AIVGMs and also assesses the judging capabilities of 13 LMMs. The authors further provide preliminary validation through human preferences, external leaderboards, and ablation studies.
Critical weaknesses
Limited diagnostic value of the aggregate rankings. The paper mainly reports aggregate preferences and overall rankings, with limited breakdowns by task attributes or rubric criteria. The aggregate rankings alone make it difficult to identify the specific capabilities on which each model performs poorly, limiting the benchmark’s usefulness for guiding targeted model improvements. The benchmark coverage and leaderboard reliability require further justification. RAVEN-Eval contains 250 tasks, and Figure 3 shows that the tasks are distributed unevenly across categories. A smaller but information-dense benchmark can still be valuable, but the paper should better justify its task scale and distribution. The human evaluation covers 50 tasks and provides preliminary evidence that RAVEN-Eval rankings align with expert preferences within the evaluated setting. However, the sample size and the details reported are insufficient to support strong claims about overall reliability. In addition, although the paper reports the correlations between each of 13 LMM judges and the human reference rankings, it provides limited analysis of the agreement among the three judges used for the main leaderboard and of the leaderboard’s sensitivity to judge selection.
Questions for authors
Could you report model performance broken down by task attributes or specific rubric criteria? Would these analyses change the aggregate rankings or reveal capability differences that are not captured by the overall scores? What is the rationale for the 250 tasks and the category distribution shown in Figure 3? Do the leaderboard correlations and model rankings remain stable when tasks are reduced, resampled, or rebalanced? For GPT-5.4, Claude Sonnet 4.6, and Gemini, the judges used for the main leaderboard, could you report their inter-judge agreement and analyze whether the model rankings remain stable when the judges are changed? The human evaluation covers only 50 tasks. Could you report confidence intervals for the correlation with the human rankings, specify the bootstrap resampling unit, and provide results separately for T2V, KFT, and FLT? Given that the human evaluation covers only a subset of the benchmark, to what extent do these results support generalizing the observed human alignment to the full task set?
3 comments
Rebuttal for reviewer VvuY(P2)
3. Judge composition and direct inter-judge agreement
We use the saved judgments from the same 50 tasks used in the human study (30 T2V, 10 KFT, and 10 FLT). In the tables below, G is GPT-5.4, C is Claude Sonnet 4.6, and M is Gemini 3.1 Flash-Lite.
First, we compare every pair of judges. A judge's answer is considered stable if it picks the same winning video after A and B switch positions, or if both answers are ties. We calculate Cohen's kappa, a measure of how closely two judges agree beyond chance, using video pairs for which both judges gave stable answers. The percentage in parentheses shows what fraction of all video pairs met that condition. The denominator includes pairs without a stable answer.
| Two judges compared | T2V: kappa (usable pairs) | KFT: kappa (usable pairs) | FLT: kappa (usable pairs) |
|---|---|---|---|
| G and C | 0.741 (56.9%) | 0.749 (58.1%) | 0.797 (55.7%) |
| G and M | 0.722 (45.9%) | 0.759 (42.6%) | 0.795 (31.9%) |
| C and M | 0.732 (40.9%) | 0.730 (41.7%) | 0.815 (29.7%) |
Next, we remove one judge at a time, calculate the generator rankings using the remaining two judges, and compare those rankings with the human rankings. The table reports ordinary SRCC.
| Judges used | T2V SRCC with humans | KFT SRCC with humans | FLT SRCC with humans |
|---|---|---|---|
| All three | 0.940 | 0.949 | 0.767 |
| G and C | 0.910 | 0.947 | 0.820 |
| G and M | 0.892 | 0.934 | 0.833 |
| C and M | 0.925 | 0.923 | 0.644 |
We also compare each two-judge ranking directly with the ranking from all three judges. Across the three choices of judge pair, SRCC ranges from 0.950 to 0.973 for T2V, 0.962 to 0.986 for KFT, and 0.881 to 0.969 for FLT. The T2V and KFT rankings are particularly consistent. FLT changes more depending on which judges are included.
4. Confidence intervals and the resampling unit
We use the original 50 human-rated tasks, together with their saved videos, model judgments, and human votes. The paper reports bootstrap-stabilized correlations. To show the uncertainty in the agreement with human rankings, the table below gives ordinary SRCC and a 95% confidence interval for judging with a rubric (RG) and without a rubric (NR).
| Task type / number of tasks | With rubric: SRCC [95% interval] | Without rubric: SRCC [95% interval] | RG improvement [95% interval] |
|---|---|---|---|
| T2V / 30 | 0.940 [0.895, 0.955] | 0.836 [0.765, 0.865] | +0.104 [0.054, 0.165] |
| KFT / 10 | 0.949 [0.875, 0.956] | 0.860 [0.737, 0.896] | +0.089 [0.026, 0.174] |
| FLT / 10 | 0.767 [0.600, 0.899] | 0.692 [0.521, 0.811] | +0.075 [−0.128, 0.281] |
We calculate these intervals by repeating the analysis 2,000 times. Each time, we randomly select whole tasks with replacement, separately for T2V, KFT, and FLT. All human votes, model outputs, and judge scores belonging to a selected task stay together. We then calculate both rankings and their correlation again.
The rubric improves the measured correlation in all three task types. The 95% interval for the improvement is above zero for T2V and KFT. The estimate for FLT is less precise because the human study has only ten FLT tasks.
5. Scope of the reliability claim
The 30/10/10 human-study split follows the same proportions as the full 150/50/50 benchmark. The human comparisons directly measure agreement on those 50 tasks. The analyses using scores from a larger set of tasks test how stable the generator rankings are when the task sample changes. We designed the additional checks to fit the two-week rebuttal period and will expand the human study in a later benchmark release.
As the rebuttal period draws to a close, we would be sincerely grateful for any further feedback you may have on our response. In particular, we would appreciate knowing whether our response has addressed your concerns or whether any points would benefit from further clarification. Thank you again for taking the time to review our work and for your thoughtful comments.
Rebuttal for reviewer VvuY(P1)
We thank the reviewer for the suggestions. We have made every effort to address all reviewers' comments. Conducting the experiments reported below and carefully checking their results required us substantial time, and we sincerely appreciate your patience.
1. Capability-specific diagnostics already present in the appendix
We agree that a single overall ranking is less useful than knowing which tasks each model handles well. Table 7 in appendix already ranks models by task attributes. The table below gives three examples from Tables 1 and 7.
S2 covers complex scenes with many spatial details, D1 covers complex motion and interactions, and K1 covers tasks that require specialized knowledge. There are 90 S2 tasks, 108 D1 tasks, and 69 K1 tasks. One task can belong to more than one group. We also mark 33 T2V, 13 KFT, and 8 FLT tasks as reasoning tasks.
Veo 3.1 Lite ranks first on S2 but eighth overall. LTX 2.3 Fast ranks third on K1 but fifteenth overall, while Wan 2.7 ranks fourth on S2 but twelfth overall. We will make these differences, together with the number of tasks in each group, more visible in the paper so readers can identify specific strengths and weaknesses.
2. Benchmark size and uneven category coverage
We chose 150 T2V, 50 KFT, and 50 FLT tasks to cover all three generation settings while keeping the evaluation cost manageable. The six T2V task categories contain 30, 30, 21, 21, 30, and 18 tasks. Their sizes differ because different numbers of tasks passed our construction checks. The final distribution reflects the available eligible tasks and the evaluation budget.
To test whether task count affects the rankings, we reuse the scores already collected for the original 4,669 videos. We use only tasks for which every evaluated generator has a recorded score. This gives 125 T2V tasks for 20 generators, 47 KFT tasks for 20 generators, and 47 FLT tasks for 14 generators.
We repeat the following test 500 times.
We measure ranking similarity using Spearman rank correlation (SRCC). A value of 1 means that the two rankings are identical. The ranges below contain the middle 95% of the 500 repeated tests.
We also test what happens when all six T2V categories receive equal weight, regardless of how many tasks they contain. Compared with the original ranking, the resulting ranking has SRCC 0.997, no generator moves by more than one position, and the same five generators remain in the top five.
Finally, we compare the results with a fixed external reference ranking(same as the Artificial Analysis ranking in the main paper). We keep the reference and the set of shared generators unchanged, using 18 models for T2V and 12 models for the combined KFT/FLT (I2V) results. The correlation changes from 0.889 with all eligible T2V tasks to a median of 0.882 with half the tasks. For I2V, it changes from 0.930 to 0.916. Across the 500 tests, the half-task correlations range from 0.843 to 0.905 for T2V and 0.853 to 0.965 for I2V. Overall, the rankings change little when we reduce or rebalance the tasks. All of these calculations reuse existing scores and require no new video generation or model judging.