acceptodds
Under review as a conference paper at ICLR 2027

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

Abstract

AI video generation has entered widespread commercial use. As a result, quality differences among videos produced by state-of-the-art AI video generation models (AIVGMs) have become increasingly difficult to discern using conventional evaluation criteria, such as visual fidelity and semantic instruction following. Meanwhile, human evaluation now requires more expertise and sustained attention, substantially increasing annotation costs. This calls for automated evaluation that can reliably distinguish fine-grained differences among advanced AIVGMs with minimal human intervention. To address this challenge, we present RAVEN-Eval, a rubric-guided automated evaluation framework for AIVGMs, built primarily on the LMM-as-a-judge paradigm. Through an automatic task curation and quality-filtering pipeline, RAVEN-Eval curates 150 text-to-video (T2V) tasks and 100 image-to-video (I2V) tasks, and systematically collects more than 4,500 AIGVs. At its core, RAVEN-Eval adopts rubric-guided automated LMM preference judgement, in which LMM judges conduct pairwise comparisons according to task-specific rubrics. It further introduces an anchor-based model insertion approach to reduce the evaluation cost of incorporating new models. Finally, we evaluate 20 high-performance AIVGMs, as well as the judging capabilities of 13 LMM judges, and establish the RAVEN-Eval Leaderboards. Overall, RAVEN-Eval paves a scalable path for automatic and trustworthy evaluation of rapidly evolving AIVGMs.

Primary area:
datasets and benchmarks
Keywords:
AI Video Generation Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VvuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RAVEN-Eval, a rubric-guided LMM-as-a-judge framework for evaluating text-to-video and image-to-video generation models. It constructs 250 tasks and 4,669 generated videos, uses task-specific rubrics for order-reversed pairwise judgments with ties, and estimates model scores with a Davidson-style model. The paper also proposes anchor-based insertion of new models and reports leaderboards for 20 video generation models and 13 LMM judges. Validation includes comparisons with human rankings on 50 tasks, external leaderboards, rubric ablations, and simulated model insertion experiments.

Critical strengths

The paper addresses a practically important problem: evaluating video generation models is costly. It proposes using LMMs as judges and presents an end-to-end pipeline covering task construction, rubric-guided pairwise evaluation, and incremental model insertion. The two-level rubric provides criteria for both task-specific core requirements and general video quality, making the basis for judgments more explicit. The evaluation framework covers 20 AIVGMs and also assesses the judging capabilities of 13 LMMs. The authors further provide preliminary validation through human preferences, external leaderboards, and ablation studies.

Critical weaknesses

Limited diagnostic value of the aggregate rankings. The paper mainly reports aggregate preferences and overall rankings, with limited breakdowns by task attributes or rubric criteria. The aggregate rankings alone make it difficult to identify the specific capabilities on which each model performs poorly, limiting the benchmark’s usefulness for guiding targeted model improvements. The benchmark coverage and leaderboard reliability require further justification. RAVEN-Eval contains 250 tasks, and Figure 3 shows that the tasks are distributed unevenly across categories. A smaller but information-dense benchmark can still be valuable, but the paper should better justify its task scale and distribution. The human evaluation covers 50 tasks and provides preliminary evidence that RAVEN-Eval rankings align with expert preferences within the evaluated setting. However, the sample size and the details reported are insufficient to support strong claims about overall reliability. In addition, although the paper reports the correlations between each of 13 LMM judges and the human reference rankings, it provides limited analysis of the agreement among the three judges used for the main leaderboard and of the leaderboard’s sensitivity to judge selection.

Questions for authors

Could you report model performance broken down by task attributes or specific rubric criteria? Would these analyses change the aggregate rankings or reveal capability differences that are not captured by the overall scores? What is the rationale for the 250 tasks and the category distribution shown in Figure 3? Do the leaderboard correlations and model rankings remain stable when tasks are reduced, resampled, or rebalanced? For GPT-5.4, Claude Sonnet 4.6, and Gemini, the judges used for the main leaderboard, could you report their inter-judge agreement and analyze whether the model rankings remain stable when the judges are changed? The human evaluation covers only 50 tasks. Could you report confidence intervals for the correlation with the human rankings, specify the bootstrap resampling unit, and provide results separately for T2V, KFT, and FLT? Given that the human evaluation covers only a subset of the benchmark, to what extent do these results support generalizing the observed human alignment to the full task set?

3 comments
AuthorsOpenReview ↗

Rebuttal for reviewer VvuY(P1)

We thank the reviewer for the suggestions. We have made every effort to address all reviewers' comments. Conducting the experiments reported below and carefully checking their results required us substantial time, and we sincerely appreciate your patience.

1. Capability-specific diagnostics already present in the appendix

We agree that a single overall ranking is less useful than knowing which tasks each model handles well. Table 7 in appendix already ranks models by task attributes. The table below gives three examples from Tables 1 and 7.

GeneratorOverall T2V rankS2 rankD1 rankK1 rank
Veo 3.1 Lite8196
LTX 2.3 Fast1511163
Wan 2.7124159

S2 covers complex scenes with many spatial details, D1 covers complex motion and interactions, and K1 covers tasks that require specialized knowledge. There are 90 S2 tasks, 108 D1 tasks, and 69 K1 tasks. One task can belong to more than one group. We also mark 33 T2V, 13 KFT, and 8 FLT tasks as reasoning tasks.

Veo 3.1 Lite ranks first on S2 but eighth overall. LTX 2.3 Fast ranks third on K1 but fifteenth overall, while Wan 2.7 ranks fourth on S2 but twelfth overall. We will make these differences, together with the number of tasks in each group, more visible in the paper so readers can identify specific strengths and weaknesses.

2. Benchmark size and uneven category coverage

We chose 150 T2V, 50 KFT, and 50 FLT tasks to cover all three generation settings while keeping the evaluation cost manageable. The six T2V task categories contain 30, 30, 21, 21, 30, and 18 tasks. Their sizes differ because different numbers of tasks passed our construction checks. The final distribution reflects the available eligible tasks and the evaluation budget.

To test whether task count affects the rankings, we reuse the scores already collected for the original 4,669 videos. We use only tasks for which every evaluated generator has a recorded score. This gives 125 T2V tasks for 20 generators, 47 KFT tasks for 20 generators, and 47 FLT tasks for 14 generators.

We repeat the following test 500 times.

  1. Randomly select about half of these tasks (63 T2V, 24 KFT, and 24 FLT). For T2V, keep the six task categories in roughly their original proportions.
  2. Recalculate each generator's score using the same rule as the paper. Take the average score across the selected tasks minus the population standard deviation of those scores.
  3. Rank the generators again and compare this ranking with the one calculated from all 125, 47, or 47 eligible tasks, respectively.

We measure ranking similarity using Spearman rank correlation (SRCC). A value of 1 means that the two rankings are identical. The ranges below contain the middle 95% of the 500 repeated tests.

Task typeTasks used for the full rankingTasks randomly selected in each testMedian SRCC with the full ranking [95% range]
T2V125630.992 [0.983, 0.998]
KFT47240.986 [0.973, 0.995]
FLT47240.974 [0.919, 0.996]

We also test what happens when all six T2V categories receive equal weight, regardless of how many tasks they contain. Compared with the original ranking, the resulting ranking has SRCC 0.997, no generator moves by more than one position, and the same five generators remain in the top five.

Finally, we compare the results with a fixed external reference ranking(same as the Artificial Analysis ranking in the main paper). We keep the reference and the set of shared generators unchanged, using 18 models for T2V and 12 models for the combined KFT/FLT (I2V) results. The correlation changes from 0.889 with all eligible T2V tasks to a median of 0.882 with half the tasks. For I2V, it changes from 0.930 to 0.916. Across the 500 tests, the half-task correlations range from 0.843 to 0.905 for T2V and 0.853 to 0.965 for I2V. Overall, the rankings change little when we reduce or rebalance the tasks. All of these calculations reuse existing scores and require no new video generation or model judging.

AuthorsOpenReview ↗

Rebuttal for reviewer VvuY(P2)

3. Judge composition and direct inter-judge agreement

We use the saved judgments from the same 50 tasks used in the human study (30 T2V, 10 KFT, and 10 FLT). In the tables below, G is GPT-5.4, C is Claude Sonnet 4.6, and M is Gemini 3.1 Flash-Lite.

First, we compare every pair of judges. A judge's answer is considered stable if it picks the same winning video after A and B switch positions, or if both answers are ties. We calculate Cohen's kappa, a measure of how closely two judges agree beyond chance, using video pairs for which both judges gave stable answers. The percentage in parentheses shows what fraction of all video pairs met that condition. The denominator includes pairs without a stable answer.

Two judges comparedT2V: kappa (usable pairs)KFT: kappa (usable pairs)FLT: kappa (usable pairs)
G and C0.741 (56.9%)0.749 (58.1%)0.797 (55.7%)
G and M0.722 (45.9%)0.759 (42.6%)0.795 (31.9%)
C and M0.732 (40.9%)0.730 (41.7%)0.815 (29.7%)

Next, we remove one judge at a time, calculate the generator rankings using the remaining two judges, and compare those rankings with the human rankings. The table reports ordinary SRCC.

Judges usedT2V SRCC with humansKFT SRCC with humansFLT SRCC with humans
All three0.9400.9490.767
G and C0.9100.9470.820
G and M0.8920.9340.833
C and M0.9250.9230.644

We also compare each two-judge ranking directly with the ranking from all three judges. Across the three choices of judge pair, SRCC ranges from 0.950 to 0.973 for T2V, 0.962 to 0.986 for KFT, and 0.881 to 0.969 for FLT. The T2V and KFT rankings are particularly consistent. FLT changes more depending on which judges are included.

4. Confidence intervals and the resampling unit

We use the original 50 human-rated tasks, together with their saved videos, model judgments, and human votes. The paper reports bootstrap-stabilized correlations. To show the uncertainty in the agreement with human rankings, the table below gives ordinary SRCC and a 95% confidence interval for judging with a rubric (RG) and without a rubric (NR).

Task type / number of tasksWith rubric: SRCC [95% interval]Without rubric: SRCC [95% interval]RG improvement [95% interval]
T2V / 300.940 [0.895, 0.955]0.836 [0.765, 0.865]+0.104 [0.054, 0.165]
KFT / 100.949 [0.875, 0.956]0.860 [0.737, 0.896]+0.089 [0.026, 0.174]
FLT / 100.767 [0.600, 0.899]0.692 [0.521, 0.811]+0.075 [−0.128, 0.281]

We calculate these intervals by repeating the analysis 2,000 times. Each time, we randomly select whole tasks with replacement, separately for T2V, KFT, and FLT. All human votes, model outputs, and judge scores belonging to a selected task stay together. We then calculate both rankings and their correlation again.

The rubric improves the measured correlation in all three task types. The 95% interval for the improvement is above zero for T2V and KFT. The estimate for FLT is less precise because the human study has only ten FLT tasks.

5. Scope of the reliability claim

The 30/10/10 human-study split follows the same proportions as the full 150/50/50 benchmark. The human comparisons directly measure agreement on those 50 tasks. The analyses using scores from a larger set of tasks test how stable the generator rankings are when the task sample changes. We designed the additional checks to fit the two-week rebuttal period and will expand the human study in a later benchmark release.

As the rebuttal period draws to a close, we would be sincerely grateful for any further feedback you may have on our response. In particular, we would appreciate knowing whether our response has addressed your concerns or whether any points would benefit from further clarification. Thank you again for taking the time to review our work and for your thoughtful comments.

Reviewer_pwUzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RAVEN-Eval, a task-centric fine-grained rubric-guided automated evaluation framework for AI VGMs. The framework curates a diverse benchmark of 250 T2V and I2V tasks with 4,500 model-generated videos, evaluating each task using a two-level rubric of primary task-specific and foundational criteria. To do pairwise video evaluations efficiently at scale, the authors employs an order-robust judging protocol and use modified Davidson MLE objective. They also introduce an anchor-based model insertion strategy that reduces evaluation costs.

Critical strengths
  1. The framework has some safeguards against common auto-rater and LMM-as-a-judge traps like dual stimulii bias (in subjective in-lab studies) and eval data variance noise.
  2. The anchor-based model insertion strategy is practical and effective (as seen from the results section).
Critical weaknesses
  1. The two-stage task filtering relies on video generators and prompts generated by GPT-5.4-mini, using its own self-dependent data evaluation loop that introduces systematic bias favoring video models that align with specific OpenAI-styled phrasing patterns.
  2. LMM judges evaluate videos by sampling frames at a coarse 1 fps, making it completely blind to transient, motion-fluidity failures like high-frequency flickering, jitter, etc that occur larger time intervals.
  3. The entire curation, filtering, and rubric-generation pipeline depends heavily on proprietary, closed-source models (like "GPT-5.5" and "GPT-5.4-mini"), compromising its reproducibility, independence, and long-term stability for the broader academic community.
Questions for authors
  1. How does the alignment of the framework change if only open-weight models like Qwen3.6-27B are used as judges, rather than proprietary models like Claude or GPT-5? Is the rubric-guided boost still as prominent?
  2. Have you experimented with higher frame rates for LMMs that do not support native video? What is the associated token cost and latency trade-off?
  3. Can the pipeline be executed end-to-end without any dependency on OpenAI APIs? What would be the expected performance drop in task curation and rubric generation if utilizing Llama-3.1-70B or other models instead?
5 comments
AuthorsOpenReview ↗

Rebuttal for reviewer pwUz (P1)

We thank the reviewer for the constructive comments. We have made every effort to address all reviewers' comments. Conducting the experiments reported below and carefully checking their results required us substantial time, and we sincerely appreciate your patience.

1. Curation dependence and potential selection bias

We first compare 15 tasks that passed our solver check with 15 that failed it. Each group contains 9 T2V, 3 KFT, and 3 FLT tasks with similar available attributes. Seedance 2 Fast and LTX 2.3 Fast each generate one new video for every task. Three people compare the full videos without knowing the model names or whether the task was retained. This gives 60 new videos and 90 human votes.

On the retained tasks, Seedance receives 31 wins, 8 ties, and 6 losses. On the rejected tasks, it receives 30 wins, 8 ties, and 7 losses. When ties count as half a win, Seedance's preference rates are 77.8% and 75.6%, respectively. The difference is +2.2 percentage points, with a 95% interval of [−4.4, 8.9]. Humans prefer Seedance in both groups in this focused comparison.

We then compare the original best-of-three video selection with choosing one of the same three saved videos at random. We use 20 tasks from the original human study (12 T2V, 4 KFT, and 4 FLT) and five generator pairs chosen in advance per task. Both selection methods use the same rules for valid and failed videos. Three people rate the actual pairs selected by each method, requiring at most 600 new human votes and no new videos. We also allow at most 2,400 LMM judging calls.

We measure how often the majority answer from the three LMM judges matches the human majority (A wins, B wins, or tie). We combine the judgments made with A and B in both presentation orders. If the judges give no clear final answer, we count the comparison as incorrect. If the three humans split their votes evenly across A, B, and tie, we record a tie.

With best-of-three selection, agreement is 85/100 with rubric guidance (RG) and 74/100 without it (NR), an improvement of 11.0 percentage points [5.0, 19.0]. With random selection, agreement is 83/100 (RG) and 72/100 (NR), again an improvement of 11.0 percentage points [5.0, 18.0]. Thus, rubric guidance helps under both video-selection methods.

2. Open-weight judging: existing evidence and the missing control

We run the same RG-versus-NR comparison with Qwen3.6-27B alone. We reuse the original videos and human rankings for 50 tasks (30 T2V, 10 KFT, and 10 FLT). Both settings receive the same frames at 4 fps, use the same generator pairs, and differ only in whether the judge sees the rubric. We include only generators with available videos and human reference scores. For each task, we use at most 40 T2V pairs, 40 KFT pairs, or 28 FLT pairs, chosen so the included generators can be ranked together. The test requires at most 7,520 judge calls, with no new video generation or human annotation.

The table reports Spearman rank correlation (SRCC) between Qwen's generator rankings and the human rankings. A larger SRCC means that the two rankings agree more closely. Brackets show 95% confidence intervals.

Task typeWith rubric: SRCC [95% interval]Without rubric: SRCC [95% interval]Improvement [95% interval]
T2V0.762 [0.713, 0.820]0.567 [0.466, 0.632]+0.195 [0.131, 0.302]
KFT0.612 [0.501, 0.756]0.435 [0.334, 0.582]+0.177 [0.029, 0.331]
FLT0.512 [0.389, 0.710]0.305 [0.204, 0.525]+0.207 [0.026, 0.356]

Rubric guidance improves Qwen's agreement with human rankings for all three task types. The paper reports 0.872/0.903/0.821 for the three proprietary judges together (Table 2) and 0.423/0.347/0.535 for Qwen alone (Table 4), in T2V/KFT/FLT order. Those published numbers use the paper's bootstrap-stabilized SRCC. The table above uses ordinary SRCC from the controlled 4-fps test. We calculate the improvements before rounding.

AuthorsOpenReview ↗

Rebuttal for reviewer pwUz (P2)

3. Temporal sampling and the quality–cost trade-off

We take 200 video pairs already rated by humans, drawn from 20 tasks (12 T2V, 4 KFT, and 4 FLT). We give GPT-5.4 extracted frames at 1, 4, or 8 fps in every condition, with native-video input turned off. We keep the videos, image resolution, prompt, and all other settings the same. We also identify 100 motion-sensitive pairs from ten tasks before running the tests.

Each pair is judged in both A/B orders at all three frame rates, requiring 1,200 calls and no new human labels. All 200 pairs count toward agreement. If the judge gives no clear final answer, we count it as incorrect. We report the number of input tokens and the time taken for each request.

Frames per secondAgreement with humans, 200 pairs [95% interval]Agreement on 100 motion-sensitive pairsMedian input tokens per requestMedian / 95th-percentile response time
179.0% [73.0, 85.5]65.0%11,5204.8 / 9.2 s
489.0% [86.5, 91.5]87.0%42,11210.7 / 19.1 s
892.0% [89.5, 94.5]92.0%83,10418.1 / 30.7 s

Compared with 1 fps, agreement improves by 10.0 percentage points [4.5, 15.0] at 4 fps and 13.0 percentage points [6.5, 19.0] at 8 fps. The improvement is larger for motion-sensitive videos. More frames also increase the number of input tokens and the response time.

4. Reproducible evaluation versus fully open reconstruction

The existing benchmark can already be judged with Qwen using saved tasks, rubrics, and videos, so this evaluation requires no new OpenAI API calls. We also test open-weight models for creating the tasks and rubrics themselves. Llama-3.1-70B-Instruct writes the text, and Qwen3.6-27B interprets the visual inputs.

We use 30 inputs (18 T2V, 6 KFT, and 6 FLT). For each input, we examine both the original task and rubric and a version made with the open-weight models. Two reviewers check each version, giving 120 short reviews in total. The same reviewers resolve any disagreements.

What the reviewers checkOriginal versionOpen-weight version
The task is clear and can be carried out29/30 (96.7%)28/30 (93.3%)
The rubric covers every important task requirement28/30 (93.3%)27/30 (90.0%)

The open-weight models produce usable tasks and rubrics on nearly all 30 inputs in this test. We will release the code used to connect these models to task and rubric construction, along with the evaluation records. We plan to test a full reconstruction of the benchmark and a larger human study in a later release, when more time and annotation resources are available.

For the reported confidence intervals, we repeat the calculations 2,000 times, each time resampling whole tasks while keeping matched results together. Across these focused experiments, the new human-annotation budget is at most 690 video-pair votes, plus 60 generated videos and 120 short task/rubric reviews, consistent with the two-week rebuttal period.

As the rebuttal period draws to a close, we would be sincerely grateful for any further feedback you may have on our response. In particular, we would appreciate knowing whether our response has addressed your concerns or whether any points would benefit from further clarification. Thank you again for taking the time to review our work and for your thoughtful comments.

Reviewer_ZEhkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RAVEN-Eval, a rubric-guided framework for evaluating AI video generation models through pairwise LMM judgments. It constructs 250 tasks covering text-to-video, keyframe expansion, and first–last-frame completion, with 4,669 selected videos from 20 generators. Task-specific rubrics guide order-swapped pairwise comparisons, and a confidence-weighted, tie-aware scoring method produces model rankings. The paper also proposes an anchor-based strategy for adding new generators with fewer comparisons. Experiments with five human annotators on 50 tasks and comparisons with external leaderboards show improved ranking agreement over several baselines.

Critical strengths
  • The rubric-guided evaluation is well motivated and supported by useful experiments. The benchmark covers three video generation settings and uses task-specific criteria to assess fine-grained details that conventional metrics may overlook. Compared with pairwise judging without rubrics, the proposed method improves agreement with human rankings across T2V, KFT, and FLT tasks (Table 2). The human annotation study and detailed rubric examples also help support the evaluation design.

  • The anchor-based model insertion strategy is practical and supported by controlled comparisons. The proposed reliability-based selection outperforms random anchors under the same comparison budget. With two anchors per tier and three judges, it achieves higher agreement with human rankings on both T2V (0.854 vs. 0.832) and KFT (0.903 vs. 0.878), while reducing the number of pairwise comparisons by 18.4%.

Critical weaknesses
  • The task selection process may bias the benchmark toward certain generators. Stage 2 retains tasks only when GPT-5.4-mini rates Seedance2 Fast at least as highly as LTX 2.3 Fast, even though both models appear in the final evaluation. The paper does not examine how this filtering affects task diversity or model rankings. In addition, selecting the best of three generations using GPT-5.4-mini makes the results dependent on an automatic selector, rather than measuring single-generation performance.

  • Several implementation details appear inconsistent with the reported evaluation protocol. The supplied code appears to restrict judge identifiers and uses aggregation options that differ from the three-judge ensemble and mean-minus-standard-deviation scoring described in the paper. The judging input format and the priority assigned to rubric criteria also appear inconsistent across the manuscript and implementation. These discrepancies need clarification, as they could affect the reported rankings and the reproducibility of the main results.

Questions for authors
  • How many candidate tasks were removed by the solver-consistency filter in each category, and how does this selection affect the relative performance of Seedance2 Fast and LTX 2.3 Fast? Could the authors also assess the impact of best-of-three video selection by comparing it with random candidate selection on the human-evaluated subset?

  • Could the authors clarify the exact implementation used to produce Tables 1–4? The supplied code appears inconsistent with the paper's description of the three-judge ensemble, score aggregation, rubric priorities, and video sampling. Please explain these differences and whether they affect the reported rankings and agreement with human preferences.

3 comments
AuthorsOpenReview ↗

Rebuttal for reviewer ZEhk (P1)

We thank the reviewer for the helpful comments. We have made every effort to address all reviewers' comments, including the requested revisions and supplementary experiments. Conducting the experiments reported below and carefully checking their results required us substantial time, and we sincerely appreciate your patience.

1. Solver-conditioned task selection

The numbers at each selection step are shown below.

Task typeStarting tasksRemoved in first checkSent to solver checkRemoved by solver checkRemoved for other reasonsFinal tasks
T2V3601801801812150
KFT747679850
FLT7612648650

The first check removes tasks that do not meet our complexity or source requirements. The solver check keeps tasks for which GPT-5.4-mini rates Seedance 2 Fast at least as highly as LTX 2.3 Fast. The remaining exclusions cover repeated or unclear tasks and unsuitable reference images or video segments. We will provide a task-by-task log showing when and why each candidate was removed.

To test whether the solver check favors Seedance 2 Fast, we compare 15 tasks that passed the check with 15 that failed it. Both groups contain 9 T2V, 3 KFT, and 3 FLT tasks, selected to have similar available attributes. For every task, Seedance 2 Fast and LTX 2.3 Fast each generate one new video using seeds chosen in advance. Three human raters, who do not know the model names or which tasks passed the check, compare the full videos. This requires 60 new videos and 90 human votes.

Task groupSeedance wins / ties / lossesSeedance preference, counting ties as half a win
Passed the check31 / 8 / 677.8%
Failed the check30 / 8 / 775.6%

Seedance is preferred in both groups. Its preference rate is 2.2 percentage points higher on the tasks that passed the check. The 95% interval for this difference is [−4.4, 8.9] percentage points. We calculate the interval by resampling matched tasks within each task type 2,000 times. This gives a focused human check of the two solvers used in task selection.

2. Best-of-three selection and the evaluation target

We use 20 tasks already included in the human study (12 T2V, 4 KFT, and 4 FLT). For each task, we choose five generator pairs in advance, giving 100 comparisons. For each generator in a pair, we select a video from the same three saved candidates in two ways. We use either GPT-5.4-mini's original best-of-three choice or one random choice made with a fixed seed. Both methods follow the same rules for eligible and failed outputs.

Three human raters compare the actual video pairs chosen by each method without seeing the model names. With 100 comparisons, two selection methods, and three raters per video pair, this requires at most 600 new human votes. Both methods use saved candidates, so no new videos need to be generated.

For both selection methods, we compare the human majority decision (A wins, B wins, or tie) with the majority decision of the three LMM judges. We combine the two video presentation orders before counting agreement. If the three human votes are split one way each, we count that as a tie. If the model judges do not give a clear final answer, we count that comparison as incorrect. All 100 comparisons remain in the calculation.

Video selectionAgreement with humans: RGAgreement with humans: NRRG improvement [95% interval]
Best of three85/100 (85.0%)74/100 (74.0%)+11.0 pp [5.0, 19.0]
Random choice83/100 (83.0%)72/100 (72.0%)+11.0 pp [5.0, 18.0]

RG means the judges receive the task rubric, while NR means they do not. pp means percentage points. Under RG, random selection gives 2.0 percentage points lower agreement than best-of-three selection, with a 95% interval of [−6.0, 2.0] for the difference. The advantage of rubric guidance is 11.0 percentage points under both methods. For the intervals, we resample entire tasks 2,000 times within each task type, keeping the results of both methods together.

We designed the above small human studies to fit the two-week rebuttal period and available annotation resources. We will carry out broader human validation in a later benchmark release.

AuthorsOpenReview ↗

Rebuttal for reviewer ZEhk (P2)

3. Released implementation versus the reported protocol

The supplementary code is an early test version designed for GPT judges only. It ignores results from other judges and does not include the final step that subtracts the standard deviation from the mean score. Actually ,we used the final evaluation pipeline, with the settings below, to produce the results reported in the paper.

SettingImplementation used for Tables 1–4
Main judgesGPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Flash-Lite.
Video inputEach of the three judges receives the video directly if its interface supports video. Otherwise, it receives frames in time order at 1 frame per second.
Pairwise judgingModel names are hidden. We judge each video pair twice, swapping the positions of A and B. We then convert both answers back to the original model identities. If the same video wins in both orders, or both answers are ties, the judgment receives full confidence. Other valid answer combinations receive the lower confidence weights in Eq. (1).
Rubric priorityWhen both videos meet the basic requirements to a similar degree, the task-specific details matter most. Serious failures in meaning or visual quality are still considered.
Final scoresWe combine the pairwise results using the Davidson model in Eq. (1), which accounts for ties and judgment confidence. For each generator, the final score is the mean of its task scores minus their standard deviation, calculated separately for T2V, KFT, and FLT.

The four tables use these settings as follows.

  • Table 1: All three judges are combined. The final rankings use the mean-minus-standard-deviation scores described above.
  • Table 2: The rubric-guided (RG) and no-rubric (NR) pairwise tests use the same three judges. The RG-AS baseline averages individual video scores on a 0–5 scale. VideoScore and LOVE use their own absolute-score averaging methods.
  • Table 3: The anchor-insertion test uses one, two, or three judges, as specified in the paper. We repeat the test on ten different splits, each with 15 existing and five newly inserted generators.
  • Table 4: We test 13 LMM judges one at a time, using the same pairwise scoring method. We compare each judge's generator ranking with the human ranking using SRCC and KRCC, two measures of ranking similarity. We calculate these values with the repeated-sampling method defined in Appendix A.5.

We will correct both descriptions to match the settings actually used in the code in the next version. Tables 1–4 and their reported agreement with human preferences were calculated with the final pipeline. We will release the complete runner, rubric prompts, judge settings, scoring options, and commands used to produce each table in the next version.

As the rebuttal period draws to a close, we would be sincerely grateful for any further feedback you may have on our response. In particular, we would appreciate knowing whether our response has addressed your concerns or whether any points would benefit from further clarification. Thank you again for taking the time to review our work and for your thoughtful comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.