- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark for evaluating each stage of a short-drama generation pipeline. It evaluates a large set of production models at the individual stages and, in addition to human annotation, introduces an agent-based evaluation system that can automatically score outputs in a way that aligns with human ratings.
Critical strengths
Building on prior video-generation and short-drama benchmarks, this paper substantially increases the granularity of evaluation across production stages. It covers multiple production stages and model types, and assigns distinct and sufficiently detailed dimensions to assess how well each step performs in each aspect. These dimensions are grounded in domain experts' experience, which is highly valuable. The work also involved a substantial human annotation effort and significant time investment.
Critical weaknesses
-
Table 12 shows that scores given by agentic judge on several dimensions have near-zero or negative correlations with human ratings, yet these scores still contribute to the results in Table 2. Why retain these unreliable scores for comparing models? Do the rankings and conclusions change when these dimensions are scored by humans or excluded from the automatic aggregate? If you think the results are unreliable, why don't just remove these dimensions?
-
Models in Table 2 are evaluated on different task and sample subsets. For example, seedance-2.0 covers only animation, seedance-2.5 only animated multi-reference tasks, and veo-3.1 and wan2.7 only first/last-frame tasks as reported in Appendix D.5. How can their average scores be directly compared?
-
The upstream-replacement experiment shows lower downstream scores, but does not sufficiently demonstrate how specific errors propagate and accumulate. Could you provide a visual example showing how errors accumulate downstream when upstream quality degrades?
-
The trained judge is validated only on small held-out subsets of keyframes and shot videos. If it is intended for future evaluations, please test it on unseen generation models or held-out dramas.
-
You claim the benchmark covers the short drama production chain end to end, but all main evaluations use a fixed agent ViMax. First, why ViMax? Second, since agent design itself is important for short-drama generation, this limits the claim that the evaluation establishes about end-to-end generation systems. Third, even within this setup, would the component-model rankings remain consistent if ViMax were replaced by another agent framework?
Questions for authors
Please answer the questions in the Weaknesses part.
5 comments
Response to Reviewer g1ky (2/2)
W3: Tracing specific defects across stages. The replacement study (Fig. 3, Appendix G.5) measures how much a weaker upstream model costs downstream. The new figure for Appendix G.5 (anonymous link: https://anonymous.4open.science/api/repo/ICLR-2027-Re-B213/file/traced_defects.png) shows how such costs arise: it follows two defects from the stage where they first appear to the complete drama, marks them in the frames and gives the human score and the annotators' notes at each stage.
(a) Speaker swap (The Year of Cicadas, episode 3, shot 3). The gpt-5.5-xhigh storyboard packs three turns of dialogue and a lateral reveal into one shot, which two of the three annotators flag as chaining too many actions (executability 3.00). Given this shot and its keyframes, happyhorse-1.1 gives two of the three lines to the wrong boys (speech quality 2.00; a speaker mismatch for all three annotators), and kling-3.0-omni does the same. The complete drama reuses the shot unchanged, and the same two lines are flagged again there (overall 2.33).
(b) Archive back wall (Chronicle of Buried Injustice, episode 3). The gpt-image-2 keyframes draw the back wall of the same corridor differently in shots 1 and 13: high windows and a bright doorway in one, two small windows and a desk in the other (cross-shot scene 2.00; 'back window'). Under the first/last-frame paradigm these keyframes are the video model's inputs, so the kling-3.0-omni videos copy both versions (cross-shot scene and props 2.67; 'big window, then small'), and the other three video models given these keyframes are tagged with the same layout changes (1.67 each). The complete drama again reuses both shots unchanged.
In both chains, no later stage corrects the defect: the video model either copies it (b) or fails on the overloaded shot (a), and assembly reuses the generated shots as they are. At corpus scale, GR5 shows what such defects cost once parts are assembled: the same content scores lower as a whole than as separate parts (by 0.42 for complete dramas), in nearly every drama and for every model.
Revision: we will add this figure to Appendix G.5 of the revised PDF.
W4: Validation of the trained judge. Of the two tests you suggest, the one on unseen generation models matters most for future use, since new models keep appearing. Table 2 already includes 15 models scored automatically only (†), several released after our annotation round (e.g., seedance-2.5, wan-3.0, minimax-h3 and gpt-image-2.5), and none of their outputs were used to train the judge. Their human annotation is under way for the next version of the benchmark (GR4), which will report the trained judge's agreement with these labels and release them. The benchmark scores in Table 2 come from the agentic judge, so they do not depend on the trained judge.
W5: Choice of the generation framework. DramaChain Agent takes only the multi-stage skeleton from ViMax; the rest is rebuilt for evaluation (Sec. 3.2, Fig. 5). Unlike most open frameworks, which optimise the final video, it keeps every comparison controlled:
- Branching at the evaluated stage only. Compared models receive identical upstream artefacts and prompts.
- Fixed input settings. Each comparison shares one input paradigm and duration budget, across the three paradigms used in commercial production.
- Upstream replacement. One upstream model can be replaced while the downstream model stays fixed (Sec. 4.3).
- Every stage evaluated. Each intermediate output, from storyboards to dramas, is kept as an evaluation item.
On one script, expert review rated its output in the same band as three commercial platforms (Fig. 7).
(a) Why ViMax. Its multi-stage skeleton matches the chain commercial platforms follow (Sec. 1), and its components can be swapped, as one-stage-at-a-time comparisons require.
(b) What 'end to end' means. It refers to the production chain the benchmark covers, from storyboards to multi-episode dramas (Sec. 3.1), not to a ranking of agent frameworks. We hold the framework fixed on purpose, so that score differences are attributable to the component under test rather than to the framework.
Revision: we will make the scope of 'end to end' explicit in Sec. 1.
Thank you again for the constructive review. We hope these answers address your questions.
Follow-up on Our Response
Dear Reviewer g1ky,
Thank you again for your positive and constructive review. The revised PDF now includes the figure that traces two defects across stages (Appendix G.5, Figure R1; W3) and states the scope of 'end to end' in Sec. 1 (W5). With the discussion closing on 18 November, we would be glad to know whether our responses answer your questions, and we are happy to clarify anything further.
Response to Reviewer g1ky (1/2)
Thank you for the positive assessment, and for recognising the fine-grained, stage-by-stage evaluation of the short-drama pipeline (expert-grounded dimensions, many production models), the large human-annotated reference set and the agentic judge aligned with human ratings. Below we respond to each of your points; GRk refers to point k of our general response.
W1: Dimension selection for the composite score. Per-item agreement is low on some dimensions mainly because they are hard for humans too and annotators score them holistically (GR3), not because they carry no signal. Taking per-item PLCC below 0.1 in Tables 10–12 as low agreement (16 of 63 dimensions, including every near-zero or negative one: 6 of 20 for shot video, 7 of 10 for episode video, 2 of 6 for short dramas and 1 of 8 for storyboards; none at the keyframe stages), the rankings and conclusions do not depend on these dimensions:
(a) Scored by humans. Every stage keeps its top model, and the Table 2 order is unchanged at five stages (Kendall τ = 1.00); for shot video, one adjacent pair 0.06 apart in Table 2 swaps (τ = 0.87).
(b) Excluded. Every stage again keeps its top model, and no stage's ranking moves closer to the human one: Kendall τ with the human ranking is unchanged at five stages and falls for short dramas (1.00 → 0.87). Exclusion would, however, remove much of what the video stages measure: shot video would no longer score image quality, aesthetics, motion smoothness, audio-visual sync, sound or camera movement; episode video would keep only 3 of its 10 dimensions, losing speech, sound and performance; and short dramas would no longer score watchability or pacing.
With thresholds of 0.15 or 0.2, every stage still keeps its top model under both variants. We therefore keep these dimensions; GR3 explains why per-item agreement alone should not decide whether a dimension is kept.
W2: Comparability of the rankings. Coverage is incomplete because of provider-side content moderation and generation failures (Appendix D.5), so raw averages over different subsets are not directly comparable: Table 2 marks partial coverage with ‡, and the slice comparisons in Appendix F.1 already include only models evaluated at every level. On comparable content, the leaders do not change.
(a) Adjusted for content. Correcting each model's score for the drama episodes and paradigms it covered (fixed effects) keeps every stage's top model under both automated and human scores (GR1).
(b) Shared subset. seedance-2.0, veo-3.1 and wan2.7 meet on the first/last-frame × animation subset, which all six annotated video models cover. There, seedance-2.0 still ranks first for shot videos, episode videos and short dramas under both scores.
(c) seedance-2.5. It covers only animated multi-reference generation and is scored automatically only (†). On its 10 dramas paired with seedance-2.0, it still leads, by 0.22 (95% interval −0.17 to 0.57).