acceptodds
Under review as a conference paper at ICLR 2027

DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation

Abstract

Commercial short-drama production spans multiple stages. Most benchmarks, however, assess video generation in isolation, leaving unclear whether defects originate in the evaluated stage or propagate from earlier ones. In addition, most use manually constructed inputs that do not reflect real production conditions. We present DramaChain Bench, the first short-drama benchmark evaluating pipeline-native outputs from storyboard design to multi-episode delivery. It combines an evaluation schema with three systems for generation, human annotation and automated judging. DramaChain Dimensions defines 63 leaf dimensions across six granularities and five axes: input fidelity, internal consistency, generation plausibility, visual quality and cinematic expressiveness. From supplied scripts, DramaChain Agent generates storyboards, keyframes, shot videos and finished dramas, with its workflow and output quality benchmarked against commercial platforms. DramaChain Labeling System provides three independent assessments for each of 5,569 reference items, with spatial and temporal defect localisation where applicable. DramaChain Agentic Judge combines a master–sub-agent architecture, 29 specialist modules and iterative tool use, achieving a mean model-level PLCC of 0.918 against human scores. Using this framework, we evaluate 37 generation models: 20 text LLMs, 8 image models and 9 video models. A series of judge models trained on these annotations with supervised fine-tuning and GRPO further achieve higher agreement with human ratings than the prompted agentic judge.

Primary area:
datasets and benchmarks
Keywords:
short-drama generation, end-to-end benchmark, agentic judge, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g1kyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for evaluating each stage of a short-drama generation pipeline. It evaluates a large set of production models at the individual stages and, in addition to human annotation, introduces an agent-based evaluation system that can automatically score outputs in a way that aligns with human ratings.

Critical strengths

Building on prior video-generation and short-drama benchmarks, this paper substantially increases the granularity of evaluation across production stages. It covers multiple production stages and model types, and assigns distinct and sufficiently detailed dimensions to assess how well each step performs in each aspect. These dimensions are grounded in domain experts' experience, which is highly valuable. The work also involved a substantial human annotation effort and significant time investment.

Critical weaknesses
  1. Table 12 shows that scores given by agentic judge on several dimensions have near-zero or negative correlations with human ratings, yet these scores still contribute to the results in Table 2. Why retain these unreliable scores for comparing models? Do the rankings and conclusions change when these dimensions are scored by humans or excluded from the automatic aggregate? If you think the results are unreliable, why don't just remove these dimensions?

  2. Models in Table 2 are evaluated on different task and sample subsets. For example, seedance-2.0 covers only animation, seedance-2.5 only animated multi-reference tasks, and veo-3.1 and wan2.7 only first/last-frame tasks as reported in Appendix D.5. How can their average scores be directly compared?

  3. The upstream-replacement experiment shows lower downstream scores, but does not sufficiently demonstrate how specific errors propagate and accumulate. Could you provide a visual example showing how errors accumulate downstream when upstream quality degrades?

  4. The trained judge is validated only on small held-out subsets of keyframes and shot videos. If it is intended for future evaluations, please test it on unseen generation models or held-out dramas.

  5. You claim the benchmark covers the short drama production chain end to end, but all main evaluations use a fixed agent ViMax. First, why ViMax? Second, since agent design itself is important for short-drama generation, this limits the claim that the evaluation establishes about end-to-end generation systems. Third, even within this setup, would the component-model rankings remain consistent if ViMax were replaced by another agent framework?

Questions for authors

Please answer the questions in the Weaknesses part.

5 comments
AuthorsOpenReview ↗

Response to Reviewer g1ky (1/2)

Thank you for the positive assessment, and for recognising the fine-grained, stage-by-stage evaluation of the short-drama pipeline (expert-grounded dimensions, many production models), the large human-annotated reference set and the agentic judge aligned with human ratings. Below we respond to each of your points; GRk refers to point k of our general response.

W1: Dimension selection for the composite score. Per-item agreement is low on some dimensions mainly because they are hard for humans too and annotators score them holistically (GR3), not because they carry no signal. Taking per-item PLCC below 0.1 in Tables 10–12 as low agreement (16 of 63 dimensions, including every near-zero or negative one: 6 of 20 for shot video, 7 of 10 for episode video, 2 of 6 for short dramas and 1 of 8 for storyboards; none at the keyframe stages), the rankings and conclusions do not depend on these dimensions:

(a) Scored by humans. Every stage keeps its top model, and the Table 2 order is unchanged at five stages (Kendall τ = 1.00); for shot video, one adjacent pair 0.06 apart in Table 2 swaps (τ = 0.87).

(b) Excluded. Every stage again keeps its top model, and no stage's ranking moves closer to the human one: Kendall τ with the human ranking is unchanged at five stages and falls for short dramas (1.00 → 0.87). Exclusion would, however, remove much of what the video stages measure: shot video would no longer score image quality, aesthetics, motion smoothness, audio-visual sync, sound or camera movement; episode video would keep only 3 of its 10 dimensions, losing speech, sound and performance; and short dramas would no longer score watchability or pacing.

With thresholds of 0.15 or 0.2, every stage still keeps its top model under both variants. We therefore keep these dimensions; GR3 explains why per-item agreement alone should not decide whether a dimension is kept.

W2: Comparability of the rankings. Coverage is incomplete because of provider-side content moderation and generation failures (Appendix D.5), so raw averages over different subsets are not directly comparable: Table 2 marks partial coverage with ‡, and the slice comparisons in Appendix F.1 already include only models evaluated at every level. On comparable content, the leaders do not change.

(a) Adjusted for content. Correcting each model's score for the drama episodes and paradigms it covered (fixed effects) keeps every stage's top model under both automated and human scores (GR1).

(b) Shared subset. seedance-2.0, veo-3.1 and wan2.7 meet on the first/last-frame × animation subset, which all six annotated video models cover. There, seedance-2.0 still ranks first for shot videos, episode videos and short dramas under both scores.

(c) seedance-2.5. It covers only animated multi-reference generation and is scored automatically only (†). On its 10 dramas paired with seedance-2.0, it still leads, by 0.22 (95% interval −0.17 to 0.57).

AuthorsOpenReview ↗

Response to Reviewer g1ky (2/2)

W3: Tracing specific defects across stages. The replacement study (Fig. 3, Appendix G.5) measures how much a weaker upstream model costs downstream. The new figure for Appendix G.5 (anonymous link: https://anonymous.4open.science/api/repo/ICLR-2027-Re-B213/file/traced_defects.png) shows how such costs arise: it follows two defects from the stage where they first appear to the complete drama, marks them in the frames and gives the human score and the annotators' notes at each stage.

(a) Speaker swap (The Year of Cicadas, episode 3, shot 3). The gpt-5.5-xhigh storyboard packs three turns of dialogue and a lateral reveal into one shot, which two of the three annotators flag as chaining too many actions (executability 3.00). Given this shot and its keyframes, happyhorse-1.1 gives two of the three lines to the wrong boys (speech quality 2.00; a speaker mismatch for all three annotators), and kling-3.0-omni does the same. The complete drama reuses the shot unchanged, and the same two lines are flagged again there (overall 2.33).

(b) Archive back wall (Chronicle of Buried Injustice, episode 3). The gpt-image-2 keyframes draw the back wall of the same corridor differently in shots 1 and 13: high windows and a bright doorway in one, two small windows and a desk in the other (cross-shot scene 2.00; 'back window'). Under the first/last-frame paradigm these keyframes are the video model's inputs, so the kling-3.0-omni videos copy both versions (cross-shot scene and props 2.67; 'big window, then small'), and the other three video models given these keyframes are tagged with the same layout changes (1.67 each). The complete drama again reuses both shots unchanged.

In both chains, no later stage corrects the defect: the video model either copies it (b) or fails on the overloaded shot (a), and assembly reuses the generated shots as they are. At corpus scale, GR5 shows what such defects cost once parts are assembled: the same content scores lower as a whole than as separate parts (by 0.42 for complete dramas), in nearly every drama and for every model.

Revision: we will add this figure to Appendix G.5 of the revised PDF.

W4: Validation of the trained judge. Of the two tests you suggest, the one on unseen generation models matters most for future use, since new models keep appearing. Table 2 already includes 15 models scored automatically only (†), several released after our annotation round (e.g., seedance-2.5, wan-3.0, minimax-h3 and gpt-image-2.5), and none of their outputs were used to train the judge. Their human annotation is under way for the next version of the benchmark (GR4), which will report the trained judge's agreement with these labels and release them. The benchmark scores in Table 2 come from the agentic judge, so they do not depend on the trained judge.

W5: Choice of the generation framework. DramaChain Agent takes only the multi-stage skeleton from ViMax; the rest is rebuilt for evaluation (Sec. 3.2, Fig. 5). Unlike most open frameworks, which optimise the final video, it keeps every comparison controlled:

  • Branching at the evaluated stage only. Compared models receive identical upstream artefacts and prompts.
  • Fixed input settings. Each comparison shares one input paradigm and duration budget, across the three paradigms used in commercial production.
  • Upstream replacement. One upstream model can be replaced while the downstream model stays fixed (Sec. 4.3).
  • Every stage evaluated. Each intermediate output, from storyboards to dramas, is kept as an evaluation item.

On one script, expert review rated its output in the same band as three commercial platforms (Fig. 7).

(a) Why ViMax. Its multi-stage skeleton matches the chain commercial platforms follow (Sec. 1), and its components can be swapped, as one-stage-at-a-time comparisons require.

(b) What 'end to end' means. It refers to the production chain the benchmark covers, from storyboards to multi-episode dramas (Sec. 3.1), not to a ranking of agent frameworks. We hold the framework fixed on purpose, so that score differences are attributable to the component under test rather than to the framework.

Revision: we will make the scope of 'end to end' explicit in Sec. 1.

Thank you again for the constructive review. We hope these answers address your questions.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer g1ky,

Thank you again for your positive and constructive review. The revised PDF now includes the figure that traces two defects across stages (Appendix G.5, Figure R1; W3) and states the scope of 'end to end' in Sec. 1 (W5). With the discussion closing on 18 November, we would be glad to know whether our responses answer your questions, and we are happy to clarify anything further.

Reviewer_o3YgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DramaChain Bench, a benchmark for short-drama generation spanning storyboards, keyframes, shot videos, episodes, and complete dramas. It combines 63 evaluation dimensions, 5,569 triply annotated items, an agentic evaluator, and trained scoring models. Experiments evaluate 37 generation models and examine cross-shot failures, upstream error propagation, and agreement with human judgments.

Critical strengths

The benchmark addresses a practical gap: good individual clips do not necessarily produce a coherent episode or drama. The authors turn this observation into a complete evaluation framework, with 63 dimensions across six granularities and three independent annotations for each of 5,569 reference items. The controlled reuse of upstream artifacts also makes the comparison more informative than simply ranking independently generated videos. Another strength is the detailed analysis of the evaluator itself. The paper reports item-level agreement, scoring bias, tool ablations, and inference time, and openly identifies perceptual dimensions where automated scoring still falls behind human judgment.

Critical weaknesses

The main concern I have is about the generalization of the trained judges. Section 4.4 states that the test items are excluded from SFT and GRPO, but it does not explain whether the split also separates dramas, episodes, characters, or shared upstream artifacts. Since many outputs are derived from the same small set of stories, an item-level split could leave closely related content on both sides. The reported advantage over the prompted evaluator therefore does not yet establish performance on unfamiliar stories or generation models.

Another issue is the comparability of the main model rankings. Appendix D.5 explicitly documents non-random missing outputs, restricted coverage, and different duration budgets, which is useful transparency. However, Table 2 still averages each model’s available items. For example, Seedance 2.5 covers only animated, multi-reference generation, while some other models cover broader settings. Replanned storyboards further change the underlying content for certain comparisons. These differences could affect the rankings independently of model quality, so a common-subset analysis is needed before interpreting the overall leaders.

The evidence for reliable automated diagnosis is also weaker than the headline model-level correlations suggest. Table 16 reports item-level PLCC of 0.203 for episode videos and 0.384 for complete dramas. Some important perceptual dimensions are weaker still: audio-visual synchronization reaches 0.075 against a human baseline of 0.448. The authors acknowledge these limitations, but they still matter for using the evaluator to identify specific failures. The localization examples are helpful, although I could not find a quantitative evaluation of the predicted defect regions or temporal intervals.

The ablation study is useful, but it does not fully isolate the source or reliability of the gains. Table 3 computes correlations over 14 model–paradigm cells, and the keyframe comparisons use arm-specific dimension coverage. This makes it difficult to tell how much of a difference comes from the evaluator architecture rather than the evaluated dimensions. The trained judge’s improvements of 0.018 and 0.051 PLCC are also reported without confidence intervals, leaving uncertainty about whether they are stable across stories and model subsets.

Finally, some details needed to reproduce the human reference are missing. Appendix B explicitly omits the full five-level rubric texts, even though these anchors define what the numerical scores mean. The annotation protocol also includes filtering based on agreement with other raters, but the exact thresholds, enabled checks, and exclusion counts are unclear. This could affect both the apparent human consistency and the measured evaluator agreement. Providing the complete rubrics and a before/after filtering analysis would make the benchmark substantially easier to audit and reuse.

Questions for authors

1.Are the trained-judge splits disjoint by drama, episode, character, and shared upstream artifacts, or only by evaluated item? Could the authors report results on held-out dramas and, separately, held-out generation models? 2.How do the main rankings change on shared stories and input paradigms under comparable duration budgets? Could the authors also report generation success rates separately, so that missing outputs are distinguishable from the quality of successful outputs? 3.Given the weaker item-level agreement on video, which diagnostic dimensions are reliable enough for practical use? Could the authors quantify defect localization and attribution accuracy against human annotations, beyond agreement on scores? 4.Could the authors repeat the keyframe ablations on identical items and dimensions, and provide uncertainty estimates for the architecture comparisons and trained-judge gains, accounting for outputs that share a drama? 5.Could the authors provide the complete five-level rubrics, exact annotation-filtering settings and exclusion counts, and human/evaluator agreement before and after consensus-based filtering?

6 comments
AuthorsOpenReview ↗

Response to Reviewer o3Yg (1/3)

Thank you for the thorough review and the concrete questions, and for recognising the practical gap we address, the triply annotated evaluation framework, the controlled reuse of upstream artefacts and the detailed analysis of the judge. Below we respond to each of your points; GRk refers to point k of our general response.

W1 / Q1: Trained-judge splits and unseen generation models. Test items are excluded from both SFT and GRPO (Sec. 4.4, Appendix D.4), and the split is by input, not by item: every test input is held out together with the outputs of all generation models. For storyboards the input is an episode script, so the sets are disjoint by episode; for keyframes and shot videos it is a shot description, so they are disjoint by shot description but not by episode.

The more direct test of generalisation is therefore on unseen generation models (GR4). Table 2 includes 15 automatically scored models (†), several released after our annotation round (e.g., seedance-2.5, wan-3.0, minimax-h3 and gpt-image-2.5), and none of their outputs were used in training. The next version of the benchmark will add their human labels and report the trained judge's agreement with them (GR4). Table 2 itself uses the agentic judge and does not depend on this split.

W2 / Q2: Comparability of the main rankings. These differences come from provider-side content moderation, generation failures and the settings each model supports (Appendix D.5), and the slice comparisons in Appendix F.1 already use only models present at every level. On comparable content, the leaders do not change.

(a) Shared stories and paradigms. Adjusting for the drama episodes and paradigms each model covered (fixed effects) keeps every stage's top model under both automated and human scores (GR1). On the first/last-frame × animation subset shared by all six annotated video models, seedance-2.0 still ranks first at every video granularity under both scores, also without veo-3.1, the only one with a different duration budget (10 s).

(b) Models with other settings. minimax-h3 (standard 15 s, the same storyboards and first/last frames) is 0.11–0.12 above kling-3.0-omni, pixverse-c1 and wan2.7 on paired shots and level with happyhorse-1.1 and seedance-2.0. wan-3.0 (30 s, replanned storyboards) has no identical shots; per episode, it scores above each annotated model. seedance-2.5 leads seedance-2.0 by 0.22 on its 10 paired dramas (95% interval −0.17 to 0.57).

(c) Success rates, counting a unit only if all its parts were generated. They are at least 99% for storyboards and single keyframes, 95% for episode keyframes and 96% for single video shots, except minimax-h3 (90%, mostly API failures). Complete dramas need every shot and are lowest: 50% for veo-3.1 and 69% for pixverse-c1 (mainly content-moderation rejections), 82% for minimax-h3 and at least 90% for the rest.

Revision: we will add these comparisons and the success rates to Appendix D.5.

AuthorsOpenReview ↗

Response to Reviewer o3Yg (2/3)

W3 / Q3: Reliability of automated diagnosis. Human and automated evaluation have different strengths. The judge scores each dimension separately against an explicit checklist, so it is strongest where defects can be checked against the script or references; annotators score more holistically (GR3) and perceive fine temporal detail, such as lip sync and motion smoothness, which a checklist alone cannot supply (Sec. 4.2). Item-level diagnosis is therefore reliable on specific dimensions, which the paper identifies (Sec. 4.2; Tables 10–12 and 20):

(a) Reliable dimensions. We call a dimension reliable when its item-level agreement is not significantly below the leave-one-out human baseline (drama-level 95% intervals), i.e. the judge does no worse than a single annotator. This holds for 44 of the 63 dimensions (28 by point estimate, as in Tables 10–12): all storyboard and episode-keyframe dimensions, 12 of 15 single-keyframe, 9 of 20 shot-video, 7 of 10 episode-video and 4 of 6 short-drama dimensions; 14 are significantly above the baseline. The other 19 fall below it and are mostly perception-heavy, such as image quality, motion smoothness and audio-visual sync (Table 20); Appendix E.3 already advises caution with the sync score.

(b) Attribution and localisation. Annotators and the judge tag defects from the same dimension-specific vocabulary (Appendix B.2, Table 6; e.g., 'lips out of sync'), and the judge must tag every score below 5 (Appendix D.1). We matched their tags on every item and dimension (exact matches pooled over items; drama-level 95% intervals within ±0.03); the closest annotator is the one whose tags best match the judge's on each item:

StageJudge vs closest annotator, F1Judge vs all three, precision / recall / F1One annotator vs the other two, F1
Storyboards0.460.56 / 0.41 / 0.470.36
Single keyframes0.500.69 / 0.41 / 0.520.45
Episode keyframes0.630.85 / 0.42 / 0.570.60
Shot videos0.490.66 / 0.39 / 0.490.46
Episode videos0.510.69 / 0.39 / 0.500.50
Short dramas0.560.84 / 0.38 / 0.530.62

Most of the judge's tags are also chosen by an annotator (precision 0.56–0.85), but it marks fewer defects than the three annotators together (recall 0.38–0.42); its F1 against all three is above the annotator reference on storyboards, single keyframes and shot videos, level on episode videos and below it on episode keyframes and short dramas. Appendix G shows single items: annotators' boxed defects (G.1, Fig. 15) and judge and human scores side by side (G.8, Fig. 21).

W4 / Q4: Ablation. The keyframe comparison differed in dimensions because the measurement-only arm can score only dimensions that have a measurement; both comparisons now use identical items and dimensions, and every row has a drama-level 95% interval.

(a) Identical items and dimensions. All arms score the same keyframes and the same shot videos. The video comparison already used the dimensions shared by all arms (Sec. 4.5); on keyframes, the five LMM-based arms, including the trained judge, share all 15 dimensions, and only the measurement-only baseline covered fewer (9). With the dimensions it does not cover scored by the LMM (second-last row), it still falls below every LMM-based arm, so the gap between architectures does not come from the evaluated dimensions.

Scoring architectureKeyframes, PLCC [95% CI]Shot video, PLCC [95% CI]
Trained judge0.955 [0.86, 0.97]0.851 [0.65, 0.92]
Full agentic judge0.937 [0.82, 0.95]0.800 [0.52, 0.90]
Without routed measurements0.903 [0.74, 0.92]0.701 [0.33, 0.84]
Without callable tools0.891 [0.72, 0.91]0.656 [0.37, 0.81]
Bare LMM judge0.880 [0.71, 0.91]0.445 [0.14, 0.66]
Measurements, LMM for uncovered dimensions0.714 [0.37, 0.84]0.578 [0.15, 0.71]
Measurements only0.677 [0.34, 0.77]0.632 [0.20, 0.73]

(b) Uncertainty of the architecture comparisons. On keyframes, the full stack's interval lies entirely above that of measurement-only scoring, and on video, adding routed measurements and tools to the LMM raises PLCC from 0.445 to 0.800. Between neighbouring variants the intervals overlap, so with 14 model–paradigm cells their order is indicative.

(c) Trained-judge gains. Its margins of 0.018 and 0.051 over the full stack lie within these intervals: the trained judge matches the full stack while needing 6–27 s per prompt (Appendix D.4).

Revision: we will add these intervals and the equal-coverage row to an appendix table referenced from Table 3. The next version of the benchmark will rerun the ablation on a larger subset that includes the newly annotated models (GR4).

AuthorsOpenReview ↗

Response to Reviewer o3Yg (3/3)

W5 / Q5: Details of the human reference. The full rubric texts were omitted from Appendix B for length, and Appendix C lists the quality checks without their thresholds; we give both below, together with the agreement before and after filtering.

(a) Rubrics. Each of the 63 rubrics gives the dimension's applicability, five anchored levels, defect tags (Appendix B.2, Table 6) and the attribution rule shown to annotators. For example, audio-visual sync (V-E2):

FieldAudio-visual sync (V-E2)
ApplicabilityShots with dialogue; otherwise N/A
5Lip motion precisely matches speech, and sound effects synchronise with on-screen actions
4Lip motion and sound effects are mostly synchronised, with only a very small, hard-to-notice mismatch
3Broadly synchronised, with an occasional slight mismatch that is not obvious without close attention
2Lip motion or sound effects are out of sync in several places, readily noticeable and harmful to viewing
1Lip motion is clearly out of sync or absent, or sound effects are misplaced
Defect tagsLips out of sync; mouth moves, no sound; sound effect misaligned; other (requires a written note)
AttributionAny score below 5 must be supported by markers on the clip, each with one or more defect tags: a frame with a box around the problem, a time span, or a text note if the problem has no single time point
Offset guideUp to 0.1 s is about full marks; 0.1–0.2 s is slight; 0.2–0.3 s is noticeable; above 0.3 s is clearly misaligned

(b) Filtering settings and exclusions. Sec. 3.3 and Appendix C define the nine checks; their default thresholds are: A1 completion time below a third of the median for the annotation type and granularity; A2 a score of 2 or below without any defect marker; A3 more than half of the dimensions marked N/A; A4 rejection in manual review; B1 one score in at least 90% of a rater's submissions; B2 at most two distinct scores over at least 20 submissions; B3 identical scores on all leaf dimensions in more than 80% of submissions; B4 negative correlation with the leave-one-out consensus; B5 a mean deviation from that consensus above 1.0 point. An item is kept while at least one rater remains. Enabling all nine checks removes 7–27% of assessments per stage (26% for storyboards, 9% for each keyframe stage, 20% for shot video, 27% for episode video and 7% for short dramas).

(c) Agreement before and after filtering. Filtering changes agreement little: across the six stages, the leave-one-out human baseline moves by −0.02 to +0.07, the judge's item-level agreement by −0.02 to +0.04 and its model-level PLCC by −0.03 to +0.03.

Revision: we will add these thresholds, the per-check exclusion counts and the before/after agreement to Appendix C. Upon acceptance, we will release the full benchmark (GR6): all generated items with their upstream artefacts, the human annotations, the complete rubrics, the evaluation code with all judge prompts and the trained judge models.

Thank you again. We hope these clarifications and the new analyses address your concerns, and we welcome further questions.

Reviewer_XCjfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AI short dramas are made in steps: script, storyboard, images, clips, and then episodes. Existing tests check each step alone with perfect inputs. This paper tests the whole chain on its real outputs. It has 63 quality checks, 5,569 items each rated by three people with defects marked, and an AI judge made of agents that use tools.

Testing 37 models shows three findings. Small early mistakes grow into big ones by the time of the final drama. Shots that look fine on their own fail when put together. Models struggle most with following instructions and staying consistent. The AI judge matches humans on text and images, but not on video.

The weak points are statistics based on only 6–9 models, a small error-growth experiment, and a judge that scores more leniently than humans.

Critical strengths
  1. It fills a real gap, and the design is right. No prior benchmark I checked combines controlled one-stage-at-a-time swaps on the pipeline’s own outputs with scoring up to multi-episode dramas.

  2. The annotation resource is rich. It has 5,569 items, each rated by three people, plus 255,925 defect tags with boxes and time intervals. Defect locations make the labels checkable and useful for training judges, and that value outlasts any leaderboard.

  3. Nice findings. Errors grow as they move down the chain. Parts that look fine on their own fail once assembled. Models differ more on the following instructions than on rendering.

  4. The analysis of its own judge is honest. The paper shows where the judge fails: video, lip-sync, and watchability.

Critical weaknesses
  1. Results are tied to one pipeline and one corpus: a single framework (ViMax), 20 scripts, and scene descriptions in Chinese only.

  2. The headline numbers are statistically fragile. The 0.918 correlation and SRCC = 1.000 are based on only 6–9 models, with no confidence intervals. At the item level, video agreement is 0.2–0.4. Serious, but fixable with bootstrap intervals and by reporting item-level numbers first.

  3. The error-propagation claim rests on thin evidence: 4 dramas, one input style, 12 configurations, one sample each, no variance. This is the most serious problem, because it is contribution #1 in the paper’s own words.

  4. Gaps in model coverage distort the rankings. For example, seedance-2.0 covers only animation, yet it anchors the “strongest chain”.

  5. Possible judge bias is never checked. GPT judges score storyboards that GPT models lead. The judge is lenient by +0.4 to +0.57. The human baseline is built to favor the judge.

  6. The judge’s novelty is overstated. DirectorBench already has a tool-using multi-agent evaluator, and EvalVerse already has a fine-tuned judge. Neither is used as a baseline.

  7. The benchmark is presented as bilingual, but it is mostly Chinese. 16 of the 20 scripts have Chinese dialogue. The 4 “English” scripts still use Chinese scene descriptions and follow the format and story tropes of a Chinese short-drama platform.

Questions for authors
  1. Does the error-growth finding hold with more dramas, all input styles, and several random seeds, with variance reported?

  2. Do the model rankings stay the same if you swap the judge models, or compare models only on the content they all completed?

  3. Can reviewers access the data and rubrics now, and how does your judge compare with DirectorBench’s and EvalVerse’s evaluators?

  4. The counts don’t add up: The per-stage counts (533 + 2,125 + 429 + 2,186 + 306 + 206) don’t match the 5,569-item reference set. Can you give a per-stage, per-model table showing how many items are in the reference set, how many were dropped by quality control, and why?

  5. Can you attach the benchmark in the supplementary material?


My main reason for the score is that there are too few scripts to support general conclusions. The benchmark is built from only 20 scripts, of which the first 3 episodes (60 total) are used for video. It's 5,569 items, all of which come from these same 20 stories, so they are not independent samples. Drama-level scores rest on about 20 stories per model, and the error-propagation study uses only 4. With content this limited, model rankings and claims of errors propagating down the chain may reflect quirks of these scripts rather than general behavior.

10 comments
AuthorsOpenReview ↗

Response to Reviewer XCjf (1/3)

Thank you for the detailed review. We appreciate that you see the benchmark as filling a real gap with the right design, the localised annotations as valuable beyond any leaderboard, the findings as informative, and our analysis of the judge as honest. Below we respond to each of your points; GRk refers to point k of our general response.

W1 / R1: Generalisability of the conclusions. We understand the concern that 20 scripts may limit how far the conclusions generalise. We therefore checked whether the conclusions depend on the particular scripts, and none of our statistics treats items from the same story as independent:

(a) Robustness to the choice of scripts. Leaving out any single drama changes no stage's top model, and the automated ranking keeps Kendall's τ ≥ 0.87 with the full ranking (≥ 0.94 for storyboards and keyframes).

(b) Dependence between items. Every interval resamples whole dramas (GR2). Once the generator and paradigm are accounted for, the story explains only 7–16% of the variance in human scores, which leaves an effective sample of about 80–200 independent items per stage, four to ten times the number of stories.

(c) Diversity and scale. The 20 full ten-episode shooting scripts span genre, period, audience channel, live action and animation, and two dialogue languages (Appendix A, Table 4), and Appendix F.1 reports results by these slices. In number of stories we match MSVBench, and on every other measure the benchmark is the largest among related benchmarks; the full annotation collection alone contains 16.8 hours of complete dramas (206, each 3.0–6.5 minutes), 6.4 hours of shot videos (2,186) and about 235,000 dimension-level human scores:

BenchmarkStories or promptsGenerators evaluatedAnnotatorsHuman-annotated items
MSVBench20 stories2010≈300
DramaBoard35 real dramas923≈120
Short-Drama-Bench50 one-sentence ideas820≈1.8k
DirectorBench80 metadata entries4 workflows14<100
EntityBench140 episodes4n.r.200
VideoWeaver285 cases93 per case200
FilmBench1,169 prompts9n.r.2,878
Ours20 ten-episode scripts37 (20 LLMs, 8 image, 9 video)5435,569 (three assessments each)

Counts other than human-annotated items are as reported in each paper (n.r.: not reported); human-annotated items are as in our Table 1.

(d) Error growth and the framework. The error-growth finding now also holds on the full corpus, beyond the four dramas (GR5). We keep the framework fixed so that score differences reflect the component under test. It is DramaChain Agent, which takes only the multi-stage skeleton from ViMax and branches only at the evaluated stage, so that compared models receive identical upstream artefacts (Sec. 3.2, Fig. 5); on one script, expert review rated its output in the same band as three commercial platforms (Fig. 7).

W2: Statistical robustness of the agreement. Most related benchmarks in Table 1 validate their evaluators at the model level: VBench-2.0 and EvalVerse correlate automated and human win ratios across models, and MSVBench and FilmBench correlate automated and human rankings over 20 models and over 9 and 7 models respectively, all four without intervals. We lead with the same model-level correlation (Sec. 4.2) so that our results can be compared with theirs, and we agree that intervals and item-level agreement matter as well:

(a) Intervals. GR2 gives 95% intervals from a drama-level bootstrap for every model-level PLCC, next to a human baseline computed the same way: the mean is 0.918 [0.82, 0.93] for the judge and 0.933 [0.86, 0.97] for one annotator against the other two. The intervals are narrow for text and images and wide for shot video ([0.56, 0.87]) and short dramas ([0.59, 0.99]), where the SRCC of 1.000 likewise rests on six models. Even so, the judge matches or exceeds a single annotator at four of the six stages.

(b) Item-level agreement. Sec. 4.2 reports it right after the 0.918, relative to the leave-one-out human baseline: at or above the baseline for text and images (2.35× for storyboards), and 0.79–0.85× of it for video, where the baseline itself is only 0.24–0.45 (Table 16, Appendix F.2; Tables 10–12 per dimension).

Revision: as you suggest, we will add these intervals to the main text, after each stage's PLCC in Table 2 and after the 0.918 in Sec. 4.2 together with its human baseline; Table 16 (Appendix F.2) will give the full set.

AuthorsOpenReview ↗

Response to Reviewer XCjf (2/3)

W3 / Q1: Error propagation. Beyond the four dramas of the controlled replacement study (Appendix G.5), we tested error growth on the full corpus and traced single defects across stages:

(a) Error growth on the full corpus (Q1). Across 20 dramas, three paradigms and 13 models (GR5, drama-level intervals), the same content scores lower as one unit than as separate parts, by 0.30 for episode keyframes, 0.24 for episode videos and 0.42 for complete dramas, in nearly every drama and for every model and paradigm; weaker models lose more. This shows where defects surface and what they cost, not which stage introduced them; the replacement study and (b) address that.

(b) Tracing single defects. The new figure for Appendix G.5 (GR5; anonymous link: https://anonymous.4open.science/api/repo/ICLR-2027-Re-B213/file/traced_defects.png) follows two defects downstream: the video generated from a storyboard that packs a three-person exchange into one shot gives lines to the wrong characters, and the episode video generated from keyframes that change the set across shots shows the same mismatch.

Revision: we will add this comparison and the figure to Appendix G.5, and summarise the comparison in one sentence in Sec. 4.3 that also states the replacement study's scope.

W4 / Q2 (second part): Ranking comparability and the strongest chain. Coverage is incomplete because of provider-side content moderation and generation failures (Appendix D.5); seedance-2.0, for instance, covers only animation (‡ in Table 2). These gaps do not change the rankings that the strongest chain rests on, which Sec. 4.2 states among human-annotated models.

(a) Adjusting each model's score for the content it covered (drama-episode and paradigm fixed effects) leaves the top model at every stage unchanged, under both automated and human scores (GR1).

(b) On content that all annotated video models completed (the first/last-frame × animation subset), seedance-2.0 still ranks first for shot videos, episode videos and short dramas, under both automated and human scores.

(c) The expanded-set leaders are scored automatically only (†), so we compared them on shared inputs. gpt-image-2.5 scores 0.09 above gpt-image-2 on 295 identical keyframe prompts. seedance-2.5 scores 0.22 above seedance-2.0 on its 10 paired short dramas (95% interval −0.17 to 0.57).

W5 / Q2 (first part): Self-preference, score bias and the human baseline. The paper does report judge bias, for every stage (Appendix D.6, Table 16); what it lacked was a direct self-preference test, which we add in (a).

(a) Self-preference and judge swap (Q2). Storyboards are not scored by a GPT judge alone: Sec. 4.1 averages a GPT judge and a Claude judge. Against models from neither family, the GPT judge favours GPT storyboards by 0.13 relative to the Claude judge (95% CI 0.07–0.19), while the Claude judge shows no preference for Claude models (−0.03, CI −0.10 to 0.05); averaging halves the effect. The two judges rank all 20 storyboard models alike (τ = 0.89, same top three). Among annotated models, the choice matters at the top: under the Claude judge alone, gpt-5.5-xhigh and claude-opus-4.8-max tie at 4.21, and humans place claude-opus-4.8-max first by 0.03.

(b) Leniency. The judge is not uniformly lenient: +0.4 to +0.57 holds for three of the six stages, while storyboards, episode video and shot video have +0.28, +0.14 and −0.35 (Table 16). Per dimension, the bias ranges from +1.6 (subject–scene integration on keyframes) to −1.7 (audio-visual sync on shot video).

(c) Human baseline. It is not built for the judge but follows the standard leave-one-out definition (Sec. 3.3): each annotator against the mean of the others. When the judge is compared with the same two-annotator means as each annotator, the picture is unchanged: text and images at or above the baseline, video below.

AuthorsOpenReview ↗

Response to Reviewer XCjf (3/3)

W6 / Q3 (second part): Relation to DirectorBench and EvalVerse. We do not claim tool use or fine-tuning as new in themselves: Sec. 2 cites both works, and Table 1 shows that both evaluate videos only. We built our own judge because the benchmark must score the whole chain, storyboards and keyframes included, on the 63 dimensions that humans annotate, which video-only evaluators cannot do. Our contribution is a judge with this coverage, validated against the triple annotations per stage and per dimension relative to a human baseline (Sec. 4.2; Tables 10–12, 16).

We agree that a baseline comparison helps, so we ran DirectorBench's released evaluator, with its configured backbone (Claude Sonnet 4.6), on our annotated episode videos, the minute-long input it targets. DirectorBench's sub-metrics are assigned to our axes by definition: lighting consistency and transition quality to C, temporal coherence to P, and narration reasonableness and background-music, video–audio and text–audio consistency to E. Values are PLCC with the three-annotator mean (item level: 237 videos; model level: six models); item-level axis values average their dimensions, as in Table 18.

AxisHuman baseline, item levelDirectorBench, item levelOurs, item levelDirectorBench, model levelOurs, model level
Composite0.240.050.200.720.94
C: internal consistency0.22−0.090.17−0.010.92
P: generation plausibility0.080.010.090.600.38
E: cinematic expressiveness0.230.110.020.580.56

Our judge agrees with the annotators clearly better on the composite and on internal consistency; scored by its own five dimensions instead, DirectorBench reaches at most 0.17 and 0.12 at item level. DirectorBench leads on cinematic expressiveness, and on generation plausibility at model level. EvalVerse has not released its judge or code.

Revision: we will add this comparison to Appendix F.2.

W7, Q3 (first part), Q4 and Q5: Clarifications.

(a) Language composition (W7). The 16/4 split follows the scripts' audience channels (Appendix A, Table 4): the domestic channels have Chinese dialogue and the overseas channels English. The four English dramas are not translated Chinese stories: they use genres typical of English-language short-drama apps (a hidden billionaire, a contract marriage, a revenge rise, a vampire romance), Western settings and casts, and English dialogue, voice-over and on-screen text. All scripts share one shooting-script format (Appendix A), so the style-balanced comparison in Appendix F.1.2 isolates dialogue language: English-dialogue dramas score at most 0.12 higher, mainly on language-dependent dimensions.

(b) Counts (Q4). As the caption of Table 16 notes, the per-stage counts describe collection and pairing, so they are not meant to sum to the reference set. They total 5,785 annotated items used in each stage's analysis (storyboards deduplicated by production unit), whereas the 5,569-item reference set keeps only items with exactly three accepted assessments (Sec. 3.3): of the 5,814 annotated items, 192, 157 of them storyboards, fell below three after 215 assessments were rejected in manual review, and 53 have a fourth.

(c) Access to data and rubrics (Q3, first part; Q5). The generated videos alone far exceed the size limit for supplementary material, so we will release the full benchmark upon acceptance (GR6): all generated items with their upstream artefacts, the human annotations, the complete rubrics, the evaluation code with all judge prompts, and the trained judge models. Until then, GR6 summarises the rubrics and quality-check settings, and Appendices B and C document every dimension's implementation, defect tags and the annotation quality checks.

We hope these answers and the new analyses address your concerns, and we are happy to discuss any point further.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all reviewers for their careful reading and constructive feedback. The reviewers agree on the motivation: evaluating each stage in isolation misses a practical problem, because good clips do not guarantee a coherent episode (XCjf, o3Yg). They also recognise our three contributions: (1) the first benchmark that evaluates a short-drama pipeline on its own outputs at every stage, from storyboards to multi-episode dramas, linking defects to their upstream context and measuring how upstream weaknesses affect downstream delivery (XCjf, o3Yg); (2) a stage-by-stage diagnosis of 37 generation models on 63 expert-grounded dimensions, which profiles each model's strengths and weaknesses instead of giving a single ranking and rests on 5,569 triply annotated items (XCjf, o3Yg, g1ky); and (3) an agentic judge validated against these annotations (mean model-level PLCC 0.918), together with a trained judge model for fast, lightweight evaluation (XCjf, o3Yg, g1ky).

Below we respond to the concerns raised by several reviewers.

GR1. Comparability of the rankings. Coverage differs across models because of provider-side content moderation, generation failures and the input settings each model supports (Appendix D.5; ‡ in Table 2). To check that these gaps do not drive the rankings, we re-estimated every model's score with drama-episode and paradigm fixed effects, which adjust for the content each model covered. The top model at every stage stays the same under both automated and human scores; scores move by at most 0.15, and the only rank changes are four swaps between adjacent models, three of them within 0.04 before adjustment. Revision: we will add the adjusted scores and the common-subset comparisons to Appendix D.5.

GR2. Statistical robustness. With six to nine models per stage, model-level intervals are necessarily wide, so we also report a model-level human baseline computed the same way (one annotator's model means against those of the other two). Model-level PLCC with 95% intervals from 2,000 drama-level bootstrap resamples (models fixed):

Stage (models)Judge vs. three-annotator meanOne annotator vs. the other two
Storyboard (9)0.936 [0.86, 0.97]0.917 [0.80, 0.98]
Single keyframe (7)0.959 [0.89, 0.98]0.979 [0.93, 1.00]
Episode keyframes (7)0.973 [0.89, 0.99]0.891 [0.63, 0.98]
Shot video (6)0.755 [0.56, 0.87]0.980 [0.93, 1.00]
Episode video (6)0.935 [0.73, 0.97]0.914 [0.73, 0.99]
Short drama (6)0.947 [0.59, 0.99]0.919 [0.65, 0.99]
Mean0.918 [0.82, 0.93]0.933 [0.86, 0.97]

The judge matches or exceeds a single annotator at four of six stages; it is slightly below on single keyframes and clearly below on shot video (Sec. 4.2). It orders 109 of 123 model pairs as humans do and picks the human top model at five stages (at the sixth, storyboards, the two leading models differ by 0.03 in human score). Leaving out any single drama changes no stage's top model. Revision: we will add these intervals to the main text, after each stage's PLCC in Table 2 and after the 0.918 in Sec. 4.2 together with its human baseline; Table 16 (Appendix F.2) will give the full set.

GR3. Dimension selection. Automated and human scores are produced differently, so absolute item-level alignment on every single dimension is not the only criterion that matters. What matters more is that the 63 dimensions, across six granularities and five axes, cover the production chain and together rank models reliably (mean model-level PLCC 0.918, Sec. 4.2). Three findings support keeping the low-agreement dimensions:

(a) They are often hard for humans too. Agreement on a dimension tracks its human reproducibility (PLCC 0.42 across the 63 dimensions), and Appendix F.2.4 already concludes that human reproducibility, not the automated score alone, should decide whether a dimension is kept.

(b) Annotators score holistically. Human dimension scores share an overall impression (mean inter-dimension correlation 0.22–0.47, against 0.03–0.29 for the judge, which scores each dimension separately), so per-dimension agreement understates the signal these dimensions carry.

(c) Removing them does not bring the rankings closer to human judgement. Taking per-item PLCC below 0.1 as low agreement (16 dimensions), excluding them leaves every stage's top model unchanged and raises no stage's Kendall τ with the human ranking (it falls for short dramas, 1.00 → 0.87), while the video stages would no longer score image quality, motion smoothness, audio-visual sync or sound (our reply to g1ky, W1).

AuthorsOpenReview ↗

General Response (2/2)

GR4. Generalisation of the trained judge. The trained judge is tested on inputs it never saw in training. Test items are excluded from both SFT and GRPO (Sec. 4.4, Appendix D.4), and the split is by input, not by item: each held-out episode script (for storyboards) or shot description (for keyframes and shot videos) is held out together with the outputs that every generation model produced from it. The trained judge is an optional fast scorer; all scores in Table 2 come from the agentic judge, so the main rankings do not depend on this split. Table 2 already includes 15 models scored automatically only (†), several released after our annotation round (e.g., seedance-2.5, wan-3.0, minimax-h3 and gpt-image-2.5); their results agree with community feedback and our internal expert review. Their human annotation is under way for the next version of the benchmark, which will report the trained judge's agreement with these labels, a test on generation models it has never seen, and release the labels. Revision: we will describe the split precisely in Appendix D.4, with a one-sentence summary in Sec. 4.4.

GR5. Error propagation. Beyond the controlled replacement study on four dramas (Fig. 3, Appendix G.5), error growth holds on the full corpus: the same parts score lower once assembled into a unit, in nearly every drama and for every model and paradigm (human scores, three-annotator means; content fixed; 95% intervals from 2,000 drama-level resamples; last column: units whose parts are all usable, ≥ 3.0, but which are unusable as a whole):

Assembly step (pairs)Change [95% CI]Dramas / models / paradigms lowerUsable parts, unusable unit
Keyframes → episode keyframes (399)−0.30 [−0.37, −0.24]17/18 · 7/7 · 2/278/154
Shot videos → episode video (306)−0.24 [−0.34, −0.16]12/12 · 6/6 · 3/365/158
Shot videos → complete drama (206)−0.42 [−0.52, −0.32]19/20 · 6/6 · 3/319/37

Weaker models lose more (complete drama: −0.15 for seedance-2.0, −0.74 for veo-3.1), in line with the widening gap between models in Appendix E.3. How specific defects propagate is shown in the new figure for Appendix G.5 (our reply to g1ky, W3; anonymous link: https://anonymous.4open.science/api/repo/ICLR-2027-Re-B213/file/traced_defects.png). Revision: we will add this table and the figure to Appendix G.5 and summarise the table in one sentence in Sec. 4.3.

GR6. Reproducibility. The full rubric texts were omitted from Appendix B for length; Appendices B and C document every dimension's implementation and defect tags, and the nine quality checks. Our reply to o3Yg (W5) gives the complete rubric of one dimension, the thresholds of all nine checks and the agreement before and after filtering: enabling all nine checks removes 7–27% of assessments per stage while changing model-level PLCC by at most 0.03. Our reply to XCjf (Q4) explains how the 5,569-item reference set is counted. Revision: we will add the thresholds, per-check exclusion counts and before/after agreement to Appendix C, and upon acceptance release the full benchmark, including all generated items with their upstream artefacts, the human annotations, the rubrics, the evaluation code with all judge prompts and the trained judge models.

AuthorsOpenReview ↗

Revised PDF Uploaded

Dear Reviewers and Area Chair,

We have uploaded a revised PDF that includes all the revisions promised in our responses, marked in blue. New tables are numbered R1–R7 and the new figure R1, so the table, figure and section numbers cited in our responses are unchanged; the main text still fits within nine pages.

RevisionLocationRelated to
95% drama-level intervals for every model-level PLCC, with a model-level human baselineTable 2 and Sec. 4.2; full set in Table 16GR2 (XCjf, o3Yg)
Coverage-adjusted scores, shared-subset comparisons and generation success ratesAppendix D.5, Tables R4–R5GR1 (XCjf, o3Yg, g1ky)
Data split of the trained judgeSec. 4.4, Appendix D.4GR4 (o3Yg, g1ky)
Ablation intervals and an equal-coverage keyframe rowAppendix D.4, Table R3 (referenced from Table 3)o3Yg W4
Error growth on the full corpus and two defects traced across stages; scope of the replacement studySec. 4.3; Appendix G.5, Table R7 and Figure R1GR5 (XCjf, g1ky)
Quality-control thresholds, per-check exclusions and agreement before and after filteringAppendix C, Tables R1–R2GR6 (o3Yg, XCjf)
Comparison with DirectorBenchAppendix F.2.6, Table R6XCjf W6
Scope of 'end to end'Sec. 1g1ky W5

As the author–reviewer discussion closes on 18 November, we would be grateful to hear whether our responses and the revision address your concerns, and we are glad to answer any remaining questions before then.

AuthorsOpenReview ↗

Summary of the Discussion Period

We thank all reviewers for their careful reading and for engaging in detail throughout the discussion. Most of the concerns raised have now been addressed, as the reviewers noted:

  • Reviewer o3Yg found the shared-subset comparisons and the additional ablations helpful, and now leans toward weak acceptance.
  • Reviewer g1ky confirmed that most of their questions have been answered.
  • Reviewer XCjf noted that the coverage-adjusted rankings, the self-preference test, the like-for-like human baseline, the DirectorBench comparison and the reconciled counts address several of their concerns, and, having read our responses in detail, has no further questions.

Main additions during the discussion

  1. Coverage-adjusted rankings with drama-episode and paradigm fixed effects; the top model is unchanged at every stage.
  2. 95% drama-level intervals for every model-level PLCC, reported against a model-level human baseline.
  3. A comparison with DirectorBench at both the item and the model level.
  4. Quality-control thresholds, per-check exclusion counts, and agreement before and after filtering.
  5. A direct self-preference test for the judges.
  6. A stratified analysis of demanding content, covering multiple characters, crowds, body action, night scenes, exterior settings and complex camera moves, with the model ranking reported within each stratum.
  7. Supplementary material added to the anonymous repository (https://anonymous.4open.science/r/ICLR-2027-Re-B213/): one complete production chain with all of its human annotations, together with the underlying artefacts.

Upon acceptance we will release the full benchmark: all generated items with their upstream artefacts, the human annotations, the complete rubrics, the evaluation code with all judge prompts, and the trained judge models.

This list covers the main items only; the complete set of revisions and clarifications is given in our per-reviewer responses. Items 1–4 are in the revised PDF, marked in blue, and our earlier comment "Revised PDF Uploaded" lists where each one lands; items 5–6 are reported in our responses and will be added to the paper. For the points that could not be supported with additional experiments within the discussion period, we will work to address them in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.