acceptodds
Under review as a conference paper at ICLR 2027

DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

Abstract

Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinematic control, audio, and cross-modal synchronization. However, evaluating such videos remains challenging, since existing benchmarks largely focus on local visual quality, short-horizon temporal consistency, or generic prompt alignment, and provide limited diagnosis of workflow failures and user-dependent preferences. We introduce DirectorBench, a personalized multi-agent diagnostic benchmark for long-form video generation. DirectorBench evaluates generated videos with respect to 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria across 5 dimensions: script, visual, audio, cross-modal, and stability. Instead of reducing quality to a single aggregate score, DirectorBench localizes checkpoint-level bottlenecks and supports profile-aware evaluation. We evaluate 4 long-form video generation workflows, 6 base LLMs, and 7 user profiles. Across workflows, DirectorBench reveals a between-unit bottleneck: transition quality averages only 0.256 and reaches 0.356 for the best workflow, while prompt-level user demand fulfillment averages 0.71. A study with 14 annotators yields case-level Spearman correlation 0.739 and 84.3% pairwise ranking accuracy; the human-rated bottleneck appears in the automatic top-three list in 96.5% of cases. Additional experiments show that profile-conditioned prompts change speech density and video characteristics. These findings highlight the importance of diagnostic and profile-aware benchmarking for agentic long-form video generation. Our code is available at https://anonymous.4open.science/r/DirectorBench-835E/.

Primary area:
datasets and benchmarks
Keywords:
evaluation, agentic video generation, personalized

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZekrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DirectorBench, a diagnostic benchmark for long-form (minute-scale, multi-shot, audio-visual) video generation produced by agentic workflows. The benchmark pairs 80 structured metadata entries (objective task requirements: story arc, shot design, camera, audio) with 7 user profiles (preference weights, taste descriptors, hard constraints), and evaluates generated videos via a multi-agent pipeline over a registry of 40 checkpoints in 5 dimensions (script, visual, audio, cross-modal, stability), with content-adaptive checkpoint activation, tool-based evidence, and confidence-aware, profile-weighted aggregation. Experiments cover 4 agentic workflows, 6 base LLMs within a fixed workflow, and profile-conditioned generation on 2 systems.

Critical strengths
  1. The problem is real and timely: existing benchmarks (VBench/VBench++, Video-Bench, EvalCrafter, T2V-CompBench) target short clips, frame-level properties, or generic prompt alignment, and do not diagnose between-shot or cross-modal failures of agentic long-form pipelines.
  2. The conceptual separation between metadata-defined requirements and profile-defined preferences (Sec. 2.1), including the explicit distinction between rescoring a fixed video and regenerating under a profile-conditioned prompt, is clean.
  3. The paper is candid about its own limitations (correlation vs. causation in RQ1, heuristic confidence, the implementation/policy gap in App. B), which is commendable.
Critical weaknesses
  1. The core methodological claim is never tested against a baseline. The paper's central contribution is the multi-agent evaluation architecture. Yet no comparison with a single strong LLM judge given the same 40-checkpoint rubric and the same inputs is reported. This comparison is admittedly absent (Sec. 3.2, App. C). For a benchmark paper whose novelty is the evaluation method, this omission is disqualifying.

  2. All results depend on a single evaluator LLM, with a direct circularity. Every automatic score in the paper is produced by GPT-5.4 which is simultaneously one of the six base LLMs being evaluated in RQ2 which is also one of the six base LLMs evaluated in RQ2, introducing a potential evaluator–candidate overlap. Judge self-preference and style bias are among the best-documented failure modes of LLM-as-judge, yet no evaluator-model ablation is provided. It is unknown whether the workflow rankings, the transition-quality bottleneck, or the "complementary specialization" pattern of RQ2 survive a different judge.

  3. The advertised mechanism does not exist in the released code. Sec. 2.3 presents the metadata-first activation policy (Table 3 in App. B) as a key design feature: required-but-missing content must be retained and penalized rather than routed away as N/A. Appendix B then concedes that the public snapshot's checkpoint filter is output-only and "does not itself guarantee retention of all metadata-required checks." The paper never states which configuration produced the reported numbers. If the main results come from output-only routing, then a claimed core contribution was not in effect, and omission of requested content may have been silently scored as not-applicable.

  4. The defining property of a "long-form" benchmark is never characterized. The paper's positioning rests on "minute-long" generation. However: (a) the distribution of requested durations across the 80 metadata entries is not reported (the sole example is 65 seconds); (b) the actual durations of the generated videos, per system, are not reported.

  5. The human study is under-specified to an extent that its numbers cannot be interpreted. Missing: number of videos rated per annotator, sampling procedure and system/quality stratification, the number of pairs behind the 84.3%, annotator expertise, elicitation method for the "human weakest criterion," and the ICC form used.

  6. Figure quality indicates insufficient manual verification, and is potentially inconsistent with the AI Use Statement. Several figures contain apparent visual noise and defects, which suggests they were produced by a generative image tool rather than standard vector-graphics software. If the figures were indeed generated with AI assistance, this appears to contradict the AI Use Statement ("AI-assisted tools were only used for language polishing"); the authors should clarify the provenance of the figures.

  7. Minor citation issues: Make-a-Video is cited twice as Singer et al. 2023a/2023b.

Questions for authors
  1. Since the multi-agent evaluation architecture is a central contribution, can the authors provide controlled comparisons of human alignment, bottleneck identification, and evaluation cost to demonstrate the added value of the multi-agent design?

  2. All LLM-based evaluation relies on GPT-5.4, which is also one of the base LLMs evaluated in RQ2. How do the authors rule out evaluator-specific bias or potential self-preference? Can they repeat the key experiments using an independent evaluator model and report whether the workflow rankings, identified bottlenecks, and base-LLM specialization patterns remain consistent?

  3. Section 2.3 introduces metadata-first checkpoint activation, while Appendix B acknowledges that the released implementation uses output-based filtering without guaranteed retention of metadata-required checkpoints. Which implementation was used to produce the reported results?

  4. Since the benchmark specifically targets long-form video generation, why are the requested and actual video durations not systematically reported? Can the authors provide duration distributions across the 80 metadata entries and the evaluated systems, including completion rates and performance breakdowns by video length, to demonstrate that the benchmark meaningfully evaluates long-range consistency?

  5. Can the authors clarify the human evaluation protocol, including the number of videos rated by each annotator, sampling and stratification procedures, the number of pairwise comparisons underlying the reported 84.3% accuracy, and the specific ICC formulation? How were the 16 human perceptual criteria mapped to the automatic evaluation metrics when computing bottleneck agreement and top-three coverage?

13 comments
AuthorsOpenReview ↗

Rebuttal Response (1/2)

We thank the reviewer for the feedback and would like to clarify several aspects of our primary contributions in response to the reviewer’s concerns. Below, we address each of the points with corresponding manuscript updates.


1. W1 & Q1: Benchmark Contribution and Multi-Agent DAG Design Rationale

DirectorBench's primary contribution is an executable diagnostic benchmark protocol—integrating structured task specifications, decoupled user profiles, and a 40-checkpoint rubric—whose multi-agent DAG is specifically designed for dimension-relevant context isolation, concurrent execution, and deferred cross-modal synthesis.

  • Benchmark Protocol Positioning (Sections 1 & 5): Rather than claiming standalone pipeline modules as isolated algorithmic inventions, DirectorBench establishes an executable diagnostic benchmark protocol for long-form agentic video generation. Its measurement validity is directly verified against human perceptual judgments (Table 1: 0.7390.739 case-level Spearman ρ\rho, 84.3%84.3\% pairwise ranking accuracy, 96.5%96.5\% top-3 bottleneck coverage) and repeated evaluations (α=0.828\alpha = 0.828, activation Jaccard =0.994= 0.994).

  • Architectural Intuition for Multi-Agent Decomposition (Section 2.4): Decomposing evaluation into a multi-agent DAG provides three concrete functional advantages over a single monolithic prompt:

  1. Dimension-Relevant Context Isolation: Each specialist agent (Script, Video, Audio, Stability) receives only the rubric instructions, task fields, and tool traces relevant to its dimension. This prevents irrelevant context from saturating the prompt and avoids premature cross-dimensional interference (e.g., visual quality biasing script scoring).

  2. Parallel Execution: Independent specialist agents execute concurrently, reducing the sequential wall-clock latency of serial multi-dimension evaluation.

  3. Deferred Joint Synthesis: Cross-modal judgments (text-video, video-audio, text-audio) and final synthesis execute only after independent specialist assessments are available, giving local scoring, relational checks, and bottleneck synthesis distinct responsibilities.

  • Manuscript Revision: Clarified our benchmark protocol positioning in Sections 1 and 5 and formalized the multi-agent decomposition rationale in Section 2.4.

2. W2 & Q2: Evaluator Bias and Cross-Judge Backbone Validation (GPT-4o vs. GPT-5.4)

Re-evaluating outputs with GPT-4o alongside GPT-5.4 yields 92.1% checkpoint-level agreement (κ=0.84\kappa = 0.84), sample-level Spearman ρ=0.93\rho = 0.93, and an unchanged ranking of the evaluated models (Kendall's τ=1.00\tau = 1.00).

  • Cross-Judge Consistency Results (Section 6): To assess sensitivity to the choice of judge backbone, we re-evaluated the outputs using GPT-4o as a second judge backbone:
Comparison Metric (GPT-4o vs. GPT-5.4)Result
Samples within [0.40, 0.60]>92%>92\% (negligible mean shift of +0.004+0.004)
Checkpoint-Level Binary Agreement92.1% (Cohen's κ=0.84\kappa = 0.84)
Sample-Level Aggregated CorrelationPearson r=0.94r = 0.94 / Spearman ρ=0.93\rho = 0.93
Relative System Ranking ConsistencyKendall's τ=1.00\tau = 1.00 (invariant ranking)
  • Observed Consistency: The comparison yields high checkpoint agreement and preserves the ranking of the evaluated models under this backbone substitution.

  • Manuscript Revision: Added the "Sensitivity to Judge Backbone" analysis in Section 6.


3. W3 & Q3: Metadata-First Activation vs. Released Code Snapshot

All reported experiments in the paper were produced with metadata-first activation enabled (Section 2.3, Section 3.2, Appendix B, Table 5).

  • Execution Configuration: In all reported experiments, checkpoints required by the task metadata remain active even when the requested content is omitted and receive a normalized score of 00, whereas genuinely inapplicable optional checks are marked N/A.

  • Snapshot Boundary Clarification: As documented in Appendix B, output-only routing is strictly a boundary of the released snapshot filter rather than the experimental configuration used to generate the paper's results.

  • Manuscript Revision: Explicitly stated the experimental configuration in Section 2.3 and Section 3.2, and formalized the policy in Appendix B (Table 5).

AuthorsOpenReview ↗

Rebuttal Response (2/2)

4. W4 & Q4: Requested and Realized Video Duration Distributions

Across the 80 metadata entries, 46 specify target durations spanning 10–210 s10\text{–}210\text{ s} (2828 entries ≥60 s\ge 60\text{ s}), and realized workflow outputs average 41.2 s41.2\text{ s} to 224.1 s224.1\text{ s}.

  • Requested Duration Distribution (Section 3.2 & Appendix I, Table 9): Of the 80 metadata entries, 46 specify a target duration (1818 below 60 s60\text{ s} and 2828 at least 60 s60\text{ s}) and 34 leave duration unspecified:
Requested Duration tt (s)Metadata EntriesRequested Duration tt (s)Metadata Entries
10≤t<1510 \le t < 15260≤t<7560 \le t < 7524
15≤t<3015 \le t < 30875≤t<9075 \le t < 901
30≤t<4530 \le t < 45490≤t≤12090 \le t \le 1202
45≤t<6045 \le t < 604t>120t > 120 (210 s210\text{ s})1
Duration Specified46Duration Unspecified34
  • Realized Duration Summaries by Workflow (Section 3.2 & Appendix I, Table 10):
WorkflowMean (s)Median (s)Min (s)Max (s)
Dreamina54.055.815.1113.6
Kling41.242.611.586.7
ViMax158.1163.444.2332.6
MovieAgent224.1231.662.7371.4
  • As noted in Appendix I, formal completion rates and length-conditional score breakdowns require output-level correspondence to specified metadata entries and a declared completion rule, and are therefore not inferred from these aggregate duration summaries alone.

  • Manuscript Revision: Added "Duration Coverage" in Section 3.2 and full duration tables in Appendix I (Tables 9–10).


5. W5 & Q5: Human Evaluation Protocol and Criterion Mapping

Fourteen annotators performed two blinded ratings per video across all 80 Dreamina videos in randomized order (160 rating records total, ICC=0.871\text{ICC} = 0.871), mapped via explicit primary (1.01.0) and auxiliary (0.50.5) weights.

  • Sampling, Stratification & Rating Protocol (Section 3.1 & Appendix A): The human validation corpus consists of 80 Dreamina-generated videos—one per metadata entry across all 7 genres (Narrative/Story: 22, Cinematic: 15, Commercial/Marketing: 12, Action: 10, Sci-Fi: 8, Educational: 7, Daily-life/Vlog: 6). Fourteen annotators rated videos on a 1–5 Likert scale across 16 perceptual criteria; each video received two blinded ratings from different annotators with randomized presentation order (160 video-level rating records, ICC=0.871\text{ICC} = 0.871).

  • 40-to-16 Criterion Projection (Appendix A.1, Tables 3–4, Equation 3): Each automatic checkpoint maps to a primary human criterion (weight 1.01.0) and auxiliary criteria (weight 0.50.5) to compute the confidence-weighted projected score H^i,d\hat{H}_{i,d} (Equation 3), from which weakest-criterion alignment and top-3 bottleneck coverage are evaluated.

  • Manuscript Revision: Updated Section 3.1 and expanded Appendix A / A.1 (Tables 3–4, Equation 3).


6. W6 & W7: Figure Provenance, AI Use Statement, and Citation Fix

  • Figure Provenance & AI Use Statement: The review does not specify which figure panels contain defects or provide verifiable evidence of generative AI image synthesis. We explicitly confirm that no generative AI image tools were used to create any figures in the manuscript; all diagrams and plots were produced via standard programmatic plotting and diagramming workflows, fully consistent with our AI Use Statement (Section 7).

  • Citation Correction: We have removed the duplicate reference entry and consolidated Make-a-Video into a single citation (Singer et al., 2023) in the References section.


We hope that these clarifications can resolve your concerns. Given these substantial manuscript updates, we respectfully hope you might consider re-evaluating our paper and overall rating. Please let us know if any further questions remain.

Thanks to the authors for the rebuttal and the manuscript revisions. While several clarifications are helpful, most of my major concerns remain unresolved:

  • Re-framing the multi-agent DAG through qualitative design intuitions (context isolation, parallel execution, and deferred synthesis) does not substitute for a controlled empirical baseline. Since the evaluation pipeline is the core methodological contribution, demonstrating its advantage over a single strong LLM judge in terms of alignment, bottleneck and cost remains essential.
  • The newly added duration statistics (Appendix I) show that only 28 of the 80 metadata entries explicitly require durations ≥ 60s, with no length-conditional performance breakdown or completion rates provided. Moreover, the human validation is restricted solely to Dreamina outputs while still omitting key protocol details (e.g., annotator expertise, the number of evaluated pairs, and the exact ICC formulation).
  • More concerningly, instead of providing the requested completion rates and length-conditional performance breakdowns, both the rebuttal and Appendix I insert a defensive disclaimer stating that such analyses "require output-level correspondence to the specified metadata entries and a declared completion rule" and cannot be inferred from the aggregate summaries. These reads like an unverified LLM-generated disclaimer rather than a substantive answer (similar to Table 6 stating that "absolute activation counts are not reconstructed from these rounded percentages").
  • I am not comfortable with the insisting claim that all diagrams were produced solely via standard programmatic plotting and diagramming workflows. Figure 1 clearly exhibits raster-level visual artifacts (e.g., typical generative noise and inconsistent background colors across icon boxes), cropping defects (e.g., the icons in "Hierarchical Rubric & Diagnosis Report"), and distorted icon details (e.g., the icons for "Personalized Prompt", "Profile-Aware Overall Score", and "Per-Checkpoint Scores (0-1)") that are characteristic of generative image models rather than vector graphics software, alongside text defects (e.g., "Muti-Agent Evaluation" in Panel 3). Simply denying the use of generative tools without replacing or fixing the figure does not resolve my concern regarding manuscript verification and consistency with the AI Use Statement.

Overall, I find the work rather rough in execution and insufficiently verified by the authors, and maintain significant reservations about its methodological soundness. In its current form, the paper is unsuitable for publication at ICLR. I encourage the authors to substantially revise the manuscript for a future submission.

Reviewer_wfM9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces DirectorBench, a personalized multi-agent diagnostic benchmark designed to evaluate long-form, agentically generated videos. Addressing the limitations of existing benchmarks that focus on short-horizon visual quality or aggregate scoring, DirectorBench assesses videos using 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria spanning script, visual, audio, cross-modal, and stability dimensions. The benchmark utilizes dynamic checkpoint activation and a multi-agent pipeline to generate localized diagnostic reports.

Critical strengths
  • The separation of objective metadata intent from subjective user preference is a new contribution.
  • The benchmark reports strong human alignment, with an 84.3% pairwise ranking accuracy and a case-level Spearman correlation of 0.739 from a study with 14 annotators.
Critical weaknesses

General:

  • The paper reads off. I am unclear why personalization appears suddenly, without a clear explanation of why it is central to diagnosing long-form video generation.
  • The dataset construction process is not new, and the dataset is relatively small. It's hard for me to see how this dataset is different from many existing and established multi-scene ones. The paper needs a more thorough comparison with benchmarks such as STBench (Zhang et al.) and MoviePrompts (Wu et al.), among others.
  • I would give some credit to the evaluation framework, but it largely follows a standard multi-agent approach: decompose the evaluation into components, prompt specialist agents, and aggregate their scores. Rubric-based evaluation is also not new; VISTA (Long et al.,), for example, has already studied it.

Commenting on the experiments:

  • The human study cannot be assessed adequately. Table 1 gives correlations, 84.3% pairwise accuracy, and 96.5% top-three bottleneck coverage, which are actually very high. However, the paper does not state the number of independently rated videos, their allocation across systems and genres, how many annotators rated each video, or uncertainty intervals, which raises questions about the reliability of the results.
  • No substantial/comparable cost baselines have been studied, including comparison against a single multimodal judge, a simpler rubric judge, or the same pipeline without tools, specialist agents, dynamic activation, and confidence weighting. Without these ablations, it's hard to see the benefits of the proposed mechanism.
  • No substantial experiments with open-source or other judge backbones.

Overall, it's hard to justify the paper's contributions and how good the results are.

Questions for authors
  • Were the published results generated with the output-only checkpoint filter? How often did requested but absent content cause a checkpoint to disappear?
  • What are the exact video and annotation counts behind every Table 1 statistic, and what rule maps 16 human criteria to automatic bottlenecks?
  • Do the principal rankings and bottlenecks persist under a single-judge baseline and after enforcing metadata-first activation?
  • Can the authors publish absolute dimension scores and uncertainty for Figure 4, and reconcile the two panels of Figure 3?
6 comments
AuthorsOpenReview ↗

Rebuttal Response (1/3)

We thank the reviewer for the detailed comments and specific questions. Below, we address each of your remarks and questions with corresponding manuscript updates.


1. General Remarks: Personalization Intuition, Dataset Comparison, and Benchmark Positioning

Long-form video creation relies heavily on human-in-the-loop creative priorities, requiring both structured task specifications and decoupled user profiles. DirectorBench's contribution is an executable diagnostic benchmark protocol for long-form agentic video generation rather than claiming standalone pipeline modules as isolated algorithmic inventions.

  • Why Personalization Is Central to Long-Form Video (Section 1): Unlike isolated short clips, minute-scale multi-shot creation involves continuous human-in-the-loop creative decisions across narrative progression, camera control, and emotional audio delivery. For the exact same requested content, different creators prioritize narrative arc, visual cinematography, or emotional synchronization differently. Retaining both objective creation requirements and a declared preference brief is therefore essential from the outset.

  • Comparison with ST-Bench, MoviePrompts, and VISTA (Section 5): In the revised Related Work (Section 5), we explicitly compare DirectorBench with ST-Bench (Zhang et al., 2025; 30 stories and 300 shot-level prompts), MoviePrompts (Wu et al., 2025; 10 script prompts with character references), and VISTA (Long et al., 2025). Because evaluating multi-shot long-form videos incurs substantially higher generation and annotation costs than short-clip benchmarks, existing suites adopt distinct counting units (stories, shot prompts, vs. complete structured tasks). DirectorBench distinguishes itself by providing 80 structured JSON creation tasks with explicit script, visual, and audio requirements, reusable decoupled user profiles, and 40 checkpoint criteria that localize workflow bottlenecks rather than collapsing quality into a single aggregate score.

  • Protocol Positioning & Cross-Judge Validation (Sections 1, 5, 6): Our core contribution is an executable diagnostic benchmark protocol tailored to long-form agentic video generation, rather than claiming individual pipeline modules as standalone algorithmic contributions. Accordingly, our validation focuses on perceptual alignment with human judgments (Table 1: 0.7390.739 case-level Spearman ρ\rho, 84.3%84.3\% pairwise ranking accuracy, 96.5%96.5\% top-3 bottleneck coverage), evaluator repeatability (α=0.828\alpha = 0.828, activation Jaccard =0.994= 0.994), and cross-judge backbone robustness: in Section 6, re-evaluating outputs with GPT-4o alongside GPT-5.4 achieves 92.1% binary checkpoint agreement (Cohen's κ=0.84\kappa = 0.84), strong sample-level correlation (Pearson r=0.94r = 0.94, Spearman ρ=0.93\rho = 0.93), and an unchanged ranking of the evaluated models (Kendall's τ=1.00\tau = 1.00).

  • Manuscript Revision: Added human-in-the-loop personalization intuition in Section 1, added explicit comparisons with ST-Bench, MoviePrompts, and VISTA in Section 5, and added the GPT-4o cross-judge sensitivity study in Section 6.

AuthorsOpenReview ↗

Rebuttal Response (2/3)

2. Q1: Metadata-First Activation vs. Output-Only Filter & Activation Rates

All published results were generated under metadata-first activation (Table 5), where required-but-absent content remains active with a normalized score of 00.

  • Experimental Configuration vs. Snapshot Boundary (Section 2.3, Section 3.2, Appendix B): We confirm that all reported experimental results use metadata-first activation (Table 5): checkpoints required by the task metadata remain active even when the requested content is omitted and receive a normalized score of 00, whereas genuinely inapplicable optional checks are marked N/A. Output-only filtering is strictly a boundary of the released snapshot filter rather than the experimental configuration.

  • Empirical Checkpoint Activation Rates (Appendix B.1, Table 6): Across the evaluated videos, rounded activation rates across the 40 checkpoints are distributed as follows:

Activation RateRegistry CountCheckpoints
100%25All unconditional checkpoints (e.g., event_chain_logic, visual_artifact_frequency, resolution_sharpness, etc.)
99%2bgm_mood_match, bgm_tempo_pacing
91%4char_clothing_consistency, char_face_consistency, character_motivation, character_matching
77%4dialogue_naturalness, speech_timing, speech_emotion_fit, speech_intelligibility
76%1bgm_volume_balance
75%1lip_sync_quality
67%1object_permanence
64%1bgm_transition_smoothness
63%1temporal_logic
Total40Complete checkpoint registry
  • Manuscript Revision: Clarified the experimental configuration in Section 2.3 and Section 3.2, and added Table 5 and Table 6 in Appendix B / B.1.

3. Q2: Exact Video/Annotation Counts and Human-to-Checkpoint Mapping Rules

The human study covers 80 Dreamina-generated videos (one per metadata entry across 7 genres), each rated independently by 2 blinded annotators in randomized order (160 rating records total, ICC=0.871\text{ICC} = 0.871).

  • Corpus & Annotator Allocation (Section 3.1 & Appendix A): Fourteen annotators evaluated 80 Dreamina-generated videos—one for each benchmark metadata entry across all 7 categories (Narrative/Story: 22, Cinematic: 15, Commercial/Marketing: 12, Action: 10, Sci-Fi: 8, Educational: 7, Daily-life/Vlog: 6) on a 1–5 Likert scale. Each video received two blinded ratings from different annotators in randomized presentation order, yielding 160 video-level rating records (ICC=0.871\text{ICC} = 0.871).

  • Explicit Mapping Rule (Appendix A.1, Tables 3–4, Equation 3): Each of the 40 automatic checkpoints maps to a primary human perceptual criterion (weight 1.01.0) and auxiliary criteria (weight 0.50.5) among the 16 human criteria (H1–H16), aggregated via confidence-weighted projection (Equation 3) to compute bottleneck alignment and top-3 coverage. The complete 40-row mapping matrix is published in Table 4 (Appendix A.1).

  • Manuscript Revision: Added exact video/rating counts and genre distributions in Section 3.1 and Appendix A, and published the complete mapping tables (Tables 3–4) and projection formula (Equation 3) in Appendix A.1.


4. Q3: Persistence of Principal Rankings and Bottlenecks

The reported experiments use metadata-first activation (Table 5). The judge-backbone comparison preserves the ranking of the evaluated models (Kendall's τ=1.00\tau = 1.00), while the human study on 80 Dreamina videos provides 84.3%84.3\% pairwise ranking accuracy and 96.5%96.5\% top-3 bottleneck coverage (Table 1).

  • Metadata-First Activation Already Enforced: As confirmed in Section 3.2 and Appendix B, all published results already enforce metadata-first activation (Table 5).

  • Cross-Judge Invariance & Human Grounding: Furthermore, Section 6 shows that replacing GPT-5.4 with GPT-4o leaves the ranking of the evaluated models unchanged (Kendall's τ=1.00\tau = 1.00) with high checkpoint-level agreement (92.1%92.1\% binary agreement, κ=0.84\kappa = 0.84). Because our benchmark protocol is directly validated against human judgments in Table 1 (0.7390.739 case-level Spearman ρ\rho, 84.3%84.3\% pairwise ranking accuracy, and 96.5%96.5\% top-3 bottleneck coverage)—and its multi-agent DAG is specifically designed for dimension-isolated context, concurrent execution, and deferred cross-modal synthesis (Section 2.4)—the study supports the complete protocol's perceptual alignment on the evaluated human-validation corpus.

AuthorsOpenReview ↗

Rebuttal Response (3/3)

5. Q4: Absolute Scores for Figure 4 and Reconciliation of Figure 3

We now publish all 40 absolute normalized checkpoint means at three-decimal precision for Figure 4 (Appendix J, Table 11) and clarify the shared corpus across both panels of Figure 3.

  • Absolute Checkpoint Scores for Figure 4 (Appendix J, Table 11): Table 11 reports all 40 absolute normalized checkpoint means across the six base LLMs. Representative contrasts highlighted in Section 4.2 include:
CheckpointTypeMin MeanMax MeanPerformance Breakdown Across Base LLMs
event_chain_logicOrdinal0.5830.750Claude Opus 4.7 highest (0.7500.750); M2.7 & Kimi lowest (0.5830.583)
motion_continuityBinary0.0000.325Opus pass rate 32.5%32.5\%, Kimi 16.3%16.3\%; Seed, GLM, GPT, M2.7 0.0%0.0\%
speech_intelligibilityOrdinal0.6250.750GLM-5.1 highest (0.7500.750); GPT-5.4 & M2.7 lowest (0.6250.625)
temporal_degradationOrdinal0.6670.875Opus highest (0.8750.875), followed by M2.7 (0.8330.833); Kimi lowest (0.6670.667)
resolution_sharpnessOrdinal0.6250.750GPT-5.4 highest (0.7500.750); M2.7 lowest (0.6250.625)
lip_sync_qualityOrdinal0.2500.400Seed highest (0.4000.400), followed by Opus (0.3500.350); GLM, M2.7, Kimi lowest (0.2500.250)
ambient_sound_matchOrdinal0.1670.375Seed & Kimi highest (0.3750.375); M2.7 lowest (0.1670.167)
  • Reconciling Figure 3 Panels: As clarified in the updated Figure 3 caption, both panels are computed over the same video corpus, with effective sample counts varying across workflows. The left panel displays the pooled mean (±\pm population std) across all four workflows, while the right heatmap annotates each workflow's absolute score inside each cell and colors the cell by its deviation (Δ\Delta) from the four-workflow cohort mean.

  • Manuscript Revision: Updated the Figure 3 caption and added Appendix J (Table 11) with all 40 three-decimal checkpoint means.


We hope that these comprehensive additions, cross-judge validations, and detailed protocol tables address all of your questions and enhance the soundness of our benchmark. Given these revisions, we respectfully hope you might reconsider the overall assessment of our work. Please let us know if any further clarifications would be helpful.

References

[Long et al., 2025] Do Xuan Long, Xingchen Wan, Hootan Nakhost, Chen-Yu Lee, Tomas Pfister, and Sercan Ö. Arık. VISTA: A test-time self-improving video generation agent, October 2025.

[Wu et al., 2025] Weijia Wu, Zeyu Zhu, and Mike Zheng Shou. Automated movie generation via multi-agent cot planning. CoRR, abs/2503.07314, 2025.

[Zhang et al., 2025] Kaiwen Zhang, Liming Jiang, Angtian Wang, Jacob Zhiyuan Fang, Tiancheng Zhi, Qing Yan, Hao Kang, Xin Lu, and Xingang Pan. StoryMem: Multi-shot long video storytelling with memory, December 2025.


Reviewer_zu5ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DirectorBench is a benchmark and multi-agent evaluation framework for diagnosing long-form video generation. It combines 80 structured metadata entries, seven user profiles, and 40 checkpoints across script, visual, audio, cross-modal, and stability dimensions. Specialist agents use model judgments and tool-derived evidence to produce scores, bottleneck lists, and diagnostic reports. The experiments compare four generation workflows, evaluate six base LLMs within a fixed workflow, and distinguish profile-dependent rescoring from profile-conditioned generation. Validation includes a study with 14 annotators and repeated automatic evaluations. The results identify cross-shot transitions and audiovisual coordination as recurring weaknesses and report encouraging agreement with human quality rankings and bottleneck judgments.

Critical strengths
  1. Useful coverage of long-form video failures. The joint assessment of narrative structure, cross-shot continuity, audio, and audiovisual coordination addresses an important evaluation need. Structured task requirements and dimension-level reports make the framework potentially useful for identifying weaknesses in assembled videos.

  2. Encouraging human alignment. The 14-annotator study reports case-level Spearman correlation of 0.739 and 96.5% top-three bottleneck coverage, alongside evaluator repeatability measurements (Table 1). These results provide preliminary support for using the reports as diagnostic shortlists.

Critical weaknesses
  1. Limited validation of personalized evaluation. Figure 5 shows score sensitivity to profile weights, while Table 2 shows changes in generated content. However, neither establishes whether profile-conditioned rankings better reflect human judgments under the corresponding preferences, leaving a central aspect of the personalized evaluation insufficiently validated.

  2. Unclear contribution of multi-agent evaluation. Table 1 reports human alignment for the full pipeline, but there is no comparison with a single judge using the same model, rubric, and available evidence. It therefore remains unclear whether specialist-agent decomposition improves scoring reliability or bottleneck localization.

Questions for authors

In Figure 5, does changing the profile only modify aggregation weights, or are the underlying checkpoint judgments also recomputed using profile-specific instructions?

2 comments
AuthorsOpenReview ↗

Rebuttal Response (1/2)

We sincerely thank the reviewer for the supportive evaluation. Below, we address your specific question and critical points with corresponding manuscript updates.


1. Q1: Recomputation of Underlying Checkpoint Judgments in Figure 5

Changing the user profile uu in Figure 5 updates the complete evaluation query and recomputes the underlying preference-sensitive checkpoint judgments using profile-specific instructions and taste descriptors, prior to applying profile-weighted aggregation.

  • Profile-Conditioned Reevaluation Mechanism: When evaluating a fixed generated video vv and metadata entry mm under a new user profile uu, the evaluation function f(m,u,v)f(m, u, v) updates the complete evaluation query. Specifically, preference-sensitive checkpoints are re-evaluated against the profile's taste descriptors and stylistic priorities before profile-weighted aggregation (Equation 2) is applied, while the generated video, task metadata, checkpoint definitions, and scoring anchors remain fixed.

  • Distinction from Generation-Side Intervention: This evaluation-time reevaluation is studied separately from generation-time intervention (v=G(g(m,u))v = \mathcal{G}(g(m, u))), where changing uu modifies the creation directive and alters the generated video content itself.

  • Manuscript Revision: To make this distinction completely explicit, we updated Section 2.1, Section 4.3, and the Figure 5 caption to state that the complete evaluation query and preference-sensitive judgments are recomputed for each profile before aggregation.


2. W1: Necessity and Empirical Validation of Personalized Evaluation

The necessity and validity of personalized evaluation are grounded in the human-in-the-loop nature of long-form video creation and jointly validated across human preference briefs (Table 1), generation-side behavioral shifts (Table 2), and evaluation-side score sensitivity (Figure 5).

  • Human-in-the-Loop Intuition for Long-Form Creation: Long-form multi-shot video creation depends heavily on human-in-the-loop creative priorities across narrative progression, camera control, and emotional delivery. Because different creators prioritize distinct quality dimensions for the exact same objective task requirements, DirectorBench retains both structured metadata and a declared user profile. User-specific criteria are studied in personalized reward-model evaluation (Ma et al., 2026), while personalized video commenting models differences in content selection and expression style (Wu et al., 2024).

  • Human Validation under Preference Briefs: In our human validation study (Section 3.1), 14 annotators rated 80 Dreamina-generated videos (two blinded ratings per video, yielding 160 rating records) including assessments under the corresponding preference briefs. DirectorBench achieves a case-level Spearman correlation of 0.739 (Kendall τ=0.705\tau = 0.705), 84.3% pairwise ranking accuracy, and 96.5% top-three bottleneck coverage against human judgments (Table 1).

  • Generation-Side and Evaluation-Side Sensitivity: Holding metadata and generation workflow fixed while varying only the profile-conditioned prompt substantially alters output characteristics—shifting ASR speech density from 24.4 to 206.7 characters/minute on Dreamina and from 67.3 to 178.7 on Kling (Table 2). Conversely, holding the generated video fixed and reevaluating it under seven user profiles shifts overall scores by more than 10 percentage points on high-sensitivity cases (Figure 5). Taken together, these findings demonstrate that generic equal-weight evaluation averages away critical preference-dependent quality variation.

  • Manuscript Revision: We expanded Section 1 to articulate the human-in-the-loop creative intuition for personalization, clarified the inclusion of preference briefs and exact annotation counts in Section 3.1, and documented the complete human-to-checkpoint mapping protocol in Appendix A / A.1 (Tables 3–4, Equation 3).

AuthorsOpenReview ↗

Rebuttal Response (2/2)

3. W2: Contribution and Design Rationale of Multi-Agent Evaluation

DirectorBench contributes an executable diagnostic benchmark protocol whose multi-agent DAG is specifically designed for dimension-isolated context, concurrent execution, and deferred cross-modal synthesis, validated end-to-end via human alignment (Table 1) and cross-judge robustness (Section 6).

  • Architectural Intuition Over Monolithic Evaluation: Decomposing evaluation into a multi-agent DAG addresses three concrete limitations of a single-prompt judge:
  1. Dimension-Relevant Context Isolation: Each specialist agent (Script, Video, Audio, Stability) receives only the rubric instructions, task fields, and tool traces relevant to its dimension. This prevents irrelevant instructions and multimodal evidence from crowding the context window and avoids premature cross-dimensional interference (e.g., visual quality biasing script scoring).

  2. Parallel Execution: Independent specialist agents run concurrently, reducing the sequential wall-clock latency of serial multi-dimension evaluation.

  3. Deferred Joint Synthesis: Cross-modal evaluation (text-video, video-audio, text-audio) and final bottleneck synthesis execute only after independent specialist assessments are available, establishing a clear division of labor between local dimension scoring and relational cross-modal diagnosis.

  • End-to-End Protocol Validation & Cross-Judge Robustness: Rather than claiming standalone pipeline modules as isolated algorithmic contributions, DirectorBench establishes an executable diagnostic benchmark protocol for long-form agentic video generation. Its measurement validity is directly verified against human rankings (84.3% pairwise accuracy) and human-rated bottlenecks (96.5% top-3 coverage) (Table 1), high evaluator repeatability (α=0.828\alpha = 0.828, activation Jaccard =0.994= 0.994), and cross-judge stability: re-evaluating outputs with GPT-4o alongside GPT-5.4 achieves 92.1% binary checkpoint agreement (κ=0.84\kappa = 0.84), sample-level Spearman ρ=0.93\rho = 0.93, and an unchanged ranking of the evaluated models (Kendall's τ=1.00\tau = 1.00) (Section 6).

  • Manuscript Revision: We clarified our benchmark protocol positioning in Sections 1 and 5, formalized the three multi-agent decomposition principles in Section 2.4, and added the cross-judge sensitivity analysis in Section 6.


We hope that these clarifications and manuscript updates thoroughly address your question and concerns. Please let us know if any further clarifications would be helpful!

References

[Ma et al., 2026] Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, and Zhe Zhao. Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization. COLM 2026.

[Wu et al., 2024] Yihan Wu, Ruihua Song, Xu Chen, Hao Jiang, Zhao Cao, and Jin Yu. Understanding Human Preferences: Towards More Personalized Video to Text Generation. Proceedings of the ACM on Web Conference 2024, pp. 3952–3963.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their constructive and insightful feedback. In the revised version of our manuscript, we have addressed the common concerns raised across the reviews. Below, we detail our responses alongside the exact updates made to the manuscript.


1. Necessity, Intuition, and Empirical Validation of Personalization

Long-form creation involves human-in-the-loop creative priorities across narrative, camera, and audio delivery, different creators value distinct dimensions for the same task. DirectorBench therefore decouples objective requirements from user preference briefs. Empirically, varying profiles recomputes preference-sensitive judgments (shifting scores by >10>10 percentage points in Figure 5), alters generation behavior (ASR speech density from 24.424.4 to 206.7206.7 chars/min in Table 2), and aligns with human judgments under matching briefs (Spearman ρ=0.739\rho = 0.739, 84.3%84.3\% pairwise accuracy).

Manuscript Revision: To address this point, we expanded Section 1 (Introduction) to establish the human-in-the-loop creative intuition for why personalization is central to long-form video diagnosis, clarified profile-conditioned query and judgment recomputation in Section 2.1, Section 4.3, and the Figure 5 caption, and explicitly documented human validation under preference briefs in Section 3.1.


2. Judge Backbone Sensitivity and Robustness Across Evaluators

To assess sensitivity to the choice of judge backbone, we re-evaluated the outputs using GPT-4o as a second judge backbone. Across the evaluated corpus, over 92% of samples fall consistently within [0.40, 0.60] under both judges with a negligible mean shift of +0.004+0.004. At the checkpoint level, GPT-4o and GPT-5.4 achieve 92.1% binary agreement (Cohen's κ=0.84\kappa = 0.84), while sample-level aggregated scores correlate strongly (Pearson r=0.94r = 0.94, Spearman ρ=0.93\rho = 0.93) and the ranking of the evaluated models remains unchanged (Kendall's τ=1.00\tau = 1.00).

Manuscript Revision: To address this point, we added a dedicated "Sensitivity to Judge Backbone" analysis in Section 6 reporting the agreement metrics, correlation statistics, and ranking invariance across judge backbones.


3. Human Validation Protocol, Blinded Ratings, and Criterion Mapping

In the human study, 14 annotators rated 80 Dreamina videos (one per entry across 7 genres) on 16 perceptual criteria (1–5 scale). Each video received two randomized blinded ratings, yielding 160 rating records (ICC=0.871\text{ICC} = 0.871), while all 40 checkpoints map to these 16 criteria via primary (1.01.0) and auxiliary (0.50.5) weights.

Manuscript Revision: To address this point, we updated Section 3.1 with the exact video and rating counts and expanded Appendix A / A.1 (Tables 3–4 and Equation 3) with genre breakdowns, metric interpretations, and the complete 40-checkpoint-to-human-criterion mapping matrix.


4. Metadata-First Activation Policy and Empirical Checkpoint Activation Rates

All principal results used metadata-first activation (scoring omitted required checkpoints as 00, while output-only routing applies solely to the released filter), with 25 unconditional checkpoints activating at 100% and conditional ones at 63%–99%.

Manuscript Revision: To address this point, we clarified the experimental activation rule in Section 2.3 and Section 3.2, formalized the metadata-first policy in Appendix B (Table 5), and reported rounded activation rates for all 40 checkpoints in Appendix B.1 (Table 6).


5. Long-Form Duration Coverage, Benchmark Positioning, and Absolute Score Breakdowns

Across the 80 entries, 46 specify durations of 10–210 s (18 < 60 s, 28 ≥ 60 s) and 34 leave duration open, with realized workflow means of 54.0 s (Dreamina), 41.2 s (Kling), 158.1 s (ViMax), and 224.1 s (MovieAgent); we also formalized our multi-agent DAG rationale, contrasted DirectorBench with ST-Bench and MoviePrompts, reconciled Figure 3, and reported all 40 absolute checkpoint scores for the six base LLMs.

Manuscript Revision: To address this point, we added duration summaries in Section 3.2 and requested/realized-duration summary statistics in Appendix I (Tables 9–10), formalized the multi-agent DAG rationale in Section 2.4, added explicit comparisons with ST-Bench, MoviePrompts, and VISTA in Section 5, updated the Figure 3 caption, and published all 40 three-decimal checkpoint scores in Appendix J (Table 11).


We believe these updates thoroughly address the raised points and strengthen the empirical and methodological foundation of our work. Please let us know if any further clarifications would be helpful.

AuthorsOpenReview ↗

Follow-up Revision

We thank the reviewers for the follow-up comments. In response, we have made the following additional revisions and provided supplementary figure-construction materials.


1. Strengthened Motivation for Personalization. We revised the Introduction to explain why human-in-the-loop interaction matters in long-horizon agentic workflows and why creative tasks require attention to individual aesthetic preferences, even under the same brief. The discussion is supported by human–agent collaboration and video co-creation research.

2. Added Controlled Evaluator Comparisons. Section 4.4 and Appendix K add four baselines on the same 80 human-annotated Dreamina videos: a holistic multimodal judge (B1), a rubric judge without tool traces (B2), a single-call judge with the same backbone, rubric, and available evidence (B3), and a flat parallel DAG without deferred cross-modal synthesis (B4). Appendix K includes the full comparison table and separate analyses of all four baselines.

3. Expanded Human-Study Reporting and Cross-System Validation. Appendix A details the 14 professional annotators’ backgrounds, rubric training, example-based calibration, and blinded rating platform. It specifies the 2,935 non-tied pairs used for pairwise accuracy and the one-way random-effects average-measures ICC(1,2)=0.871, with 95% CI [0.81, 0.91]. We additionally validate 20 shared tasks across four systems, adding 60 Kling, ViMax, and MovieAgent videos to 20 Dreamina outputs. This 80-video pool yields Spearman correlation 0.72, pairwise accuracy 83%, mean per-task four-system Kendall τ=0.97, and top-3 coverage >94%. Section 3.1 summarizes these results.

4. Added Instance-Level Duration and Length-Conditioned Analyses. Appendix I reports strict duration adherence within ±20% of the target, minimum-length satisfaction at 80% of the target, mean duration ratios, and MAPE for the 46 duration-specified tasks. It also provides quality breakdowns for requests below 60 seconds, requests of at least 60 seconds, and duration-unspecified tasks. On Dreamina outputs with realized duration ≥60 seconds, human alignment is Spearman correlation 0.748, pairwise accuracy 85.1%, and top-3 bottleneck coverage 96.4%.

5. Corrected Figure 1 Typo and Provided Supplementary Construction Evidence. We corrected the “Multi-Agent Evaluation” label in Figure 1. To document the author-assembled figure workflow, we provide the editable PowerPoint, two earlier PDF exports, 13 icon screenshots, and their correspondence manifest as supplementary material. The screenshots match the corresponding embedded PowerPoint images byte-for-byte. Both the PowerPoint and PDF files have been anonymized for double-blind review. These materials support the consistency of the figure-construction workflow with our AI Use Statement.

AuthorsOpenReview ↗

Follow-up to the Follow-up Revision

We respectfully thank the reviewer again for the follow-up comments. In response, we have made the following additional revisions and updated the corresponding materials.


1. Completed Cross-System Validation Reporting. Section 3.1 and Appendix A, Table 3 report the results on 20 shared tasks across four systems: 80 videos and 2,959 non-tied pairs out of 3,160. Results include Spearman correlation 0.72, pairwise accuracy 83% (2,456/2,959), within-case bottleneck correlation 0.618, mean per-task four-system Kendall τ=0.97, top-1 bottleneck accuracy 61.3% (49.00/80 fractional credits), and top-3 coverage 95% (76.00/80 fractional credits).

2. Specified Annotator Workload and Bottleneck Tie-Breaking Rules. Appendix A reports the workload distribution: each of the 14 annotators rated 11–12 videos in the original study and 8–9 additional videos, totaling 20 videos per annotator and 280 ratings across 140 unique videos. We define the human weakest-criterion set from the two averaged ratings and specify Fractional Tie-Credit across all 80 videos in each validation pool. Automatic top-K score ties are resolved by profile priority, confidence-weighted evidence support, and canonical criterion index, maintaining exactly K candidates.

3. Added Multi-Shot Metadata Statistics and Length-Controlled RQ1 Analyses. Appendix I reports that the 34 duration-unspecified tasks require three-act narratives and multi-shot sequences, averaging 5.6 requested shots per entry. Across all 320 generated videos, 202 outputs (63.1%) reach at least 60 seconds. New Table 15 compares workflows within a common 45–90-second realized-duration window and on outputs of at least 60 seconds, reporting sample counts, mean lengths, and diagnostic scores. These results show that workflow-specific patterns remain visible within the common window. Section 4.1 references this analysis.

4. Improved Figure 1 Icons and Updated the AI Use Statement. We replaced Figure 1 with a version using AI-assisted icon restoration to improve clarity and remove background artifacts. The AI Use Statement now explicitly discloses AI-assisted image editing for improving the icons. The updated supplementary material includes the original screenshots, restored icons, editable PowerPoint, and PDF exports. Both the PowerPoint and PDF files have been anonymized.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.