- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DirectorBench, a diagnostic benchmark for long-form (minute-scale, multi-shot, audio-visual) video generation produced by agentic workflows. The benchmark pairs 80 structured metadata entries (objective task requirements: story arc, shot design, camera, audio) with 7 user profiles (preference weights, taste descriptors, hard constraints), and evaluates generated videos via a multi-agent pipeline over a registry of 40 checkpoints in 5 dimensions (script, visual, audio, cross-modal, stability), with content-adaptive checkpoint activation, tool-based evidence, and confidence-aware, profile-weighted aggregation. Experiments cover 4 agentic workflows, 6 base LLMs within a fixed workflow, and profile-conditioned generation on 2 systems.
Critical strengths
- The problem is real and timely: existing benchmarks (VBench/VBench++, Video-Bench, EvalCrafter, T2V-CompBench) target short clips, frame-level properties, or generic prompt alignment, and do not diagnose between-shot or cross-modal failures of agentic long-form pipelines.
- The conceptual separation between metadata-defined requirements and profile-defined preferences (Sec. 2.1), including the explicit distinction between rescoring a fixed video and regenerating under a profile-conditioned prompt, is clean.
- The paper is candid about its own limitations (correlation vs. causation in RQ1, heuristic confidence, the implementation/policy gap in App. B), which is commendable.
Critical weaknesses
-
The core methodological claim is never tested against a baseline. The paper's central contribution is the multi-agent evaluation architecture. Yet no comparison with a single strong LLM judge given the same 40-checkpoint rubric and the same inputs is reported. This comparison is admittedly absent (Sec. 3.2, App. C). For a benchmark paper whose novelty is the evaluation method, this omission is disqualifying.
-
All results depend on a single evaluator LLM, with a direct circularity. Every automatic score in the paper is produced by GPT-5.4 which is simultaneously one of the six base LLMs being evaluated in RQ2 which is also one of the six base LLMs evaluated in RQ2, introducing a potential evaluator–candidate overlap. Judge self-preference and style bias are among the best-documented failure modes of LLM-as-judge, yet no evaluator-model ablation is provided. It is unknown whether the workflow rankings, the transition-quality bottleneck, or the "complementary specialization" pattern of RQ2 survive a different judge.
-
The advertised mechanism does not exist in the released code. Sec. 2.3 presents the metadata-first activation policy (Table 3 in App. B) as a key design feature: required-but-missing content must be retained and penalized rather than routed away as N/A. Appendix B then concedes that the public snapshot's checkpoint filter is output-only and "does not itself guarantee retention of all metadata-required checks." The paper never states which configuration produced the reported numbers. If the main results come from output-only routing, then a claimed core contribution was not in effect, and omission of requested content may have been silently scored as not-applicable.
-
The defining property of a "long-form" benchmark is never characterized. The paper's positioning rests on "minute-long" generation. However: (a) the distribution of requested durations across the 80 metadata entries is not reported (the sole example is 65 seconds); (b) the actual durations of the generated videos, per system, are not reported.
-
The human study is under-specified to an extent that its numbers cannot be interpreted. Missing: number of videos rated per annotator, sampling procedure and system/quality stratification, the number of pairs behind the 84.3%, annotator expertise, elicitation method for the "human weakest criterion," and the ICC form used.
-
Figure quality indicates insufficient manual verification, and is potentially inconsistent with the AI Use Statement. Several figures contain apparent visual noise and defects, which suggests they were produced by a generative image tool rather than standard vector-graphics software. If the figures were indeed generated with AI assistance, this appears to contradict the AI Use Statement ("AI-assisted tools were only used for language polishing"); the authors should clarify the provenance of the figures.
-
Minor citation issues: Make-a-Video is cited twice as Singer et al. 2023a/2023b.
Questions for authors
-
Since the multi-agent evaluation architecture is a central contribution, can the authors provide controlled comparisons of human alignment, bottleneck identification, and evaluation cost to demonstrate the added value of the multi-agent design?
-
All LLM-based evaluation relies on GPT-5.4, which is also one of the base LLMs evaluated in RQ2. How do the authors rule out evaluator-specific bias or potential self-preference? Can they repeat the key experiments using an independent evaluator model and report whether the workflow rankings, identified bottlenecks, and base-LLM specialization patterns remain consistent?
-
Section 2.3 introduces metadata-first checkpoint activation, while Appendix B acknowledges that the released implementation uses output-based filtering without guaranteed retention of metadata-required checkpoints. Which implementation was used to produce the reported results?
-
Since the benchmark specifically targets long-form video generation, why are the requested and actual video durations not systematically reported? Can the authors provide duration distributions across the 80 metadata entries and the evaluated systems, including completion rates and performance breakdowns by video length, to demonstrate that the benchmark meaningfully evaluates long-range consistency?
-
Can the authors clarify the human evaluation protocol, including the number of videos rated by each annotator, sampling and stratification procedures, the number of pairwise comparisons underlying the reported 84.3% accuracy, and the specific ICC formulation? How were the 16 human perceptual criteria mapped to the automatic evaluation metrics when computing bottleneck agreement and top-three coverage?
13 comments
Rebuttal Response (2/2)
4. W4 & Q4: Requested and Realized Video Duration Distributions
Across the 80 metadata entries, 46 specify target durations spanning ( entries ), and realized workflow outputs average to .
- Requested Duration Distribution (
Section 3.2&Appendix I,Table 9): Of the 80 metadata entries, 46 specify a target duration ( below and at least ) and 34 leave duration unspecified:
| Requested Duration (s) | Metadata Entries | Requested Duration (s) | Metadata Entries |
|---|---|---|---|
| 2 | 24 | ||
| 8 | 1 | ||
| 4 | 2 | ||
| 4 | () | 1 | |
| Duration Specified | 46 | Duration Unspecified | 34 |
- Realized Duration Summaries by Workflow (
Section 3.2&Appendix I,Table 10):
| Workflow | Mean (s) | Median (s) | Min (s) | Max (s) |
|---|---|---|---|---|
| Dreamina | 54.0 | 55.8 | 15.1 | 113.6 |
| Kling | 41.2 | 42.6 | 11.5 | 86.7 |
| ViMax | 158.1 | 163.4 | 44.2 | 332.6 |
| MovieAgent | 224.1 | 231.6 | 62.7 | 371.4 |
-
As noted in
Appendix I, formal completion rates and length-conditional score breakdowns require output-level correspondence to specified metadata entries and a declared completion rule, and are therefore not inferred from these aggregate duration summaries alone. -
Manuscript Revision: Added "Duration Coverage" in
Section 3.2and full duration tables inAppendix I(Tables 9–10).
5. W5 & Q5: Human Evaluation Protocol and Criterion Mapping
Fourteen annotators performed two blinded ratings per video across all 80 Dreamina videos in randomized order (160 rating records total, ), mapped via explicit primary () and auxiliary () weights.
-
Sampling, Stratification & Rating Protocol (
Section 3.1&Appendix A): The human validation corpus consists of 80 Dreamina-generated videos—one per metadata entry across all 7 genres (Narrative/Story: 22, Cinematic: 15, Commercial/Marketing: 12, Action: 10, Sci-Fi: 8, Educational: 7, Daily-life/Vlog: 6). Fourteen annotators rated videos on a 1–5 Likert scale across 16 perceptual criteria; each video received two blinded ratings from different annotators with randomized presentation order (160 video-level rating records, ). -
40-to-16 Criterion Projection (
Appendix A.1,Tables 3–4,Equation 3): Each automatic checkpoint maps to a primary human criterion (weight ) and auxiliary criteria (weight ) to compute the confidence-weighted projected score (Equation 3), from which weakest-criterion alignment and top-3 bottleneck coverage are evaluated. -
Manuscript Revision: Updated
Section 3.1and expandedAppendix A / A.1(Tables 3–4,Equation 3).
6. W6 & W7: Figure Provenance, AI Use Statement, and Citation Fix
-
Figure Provenance & AI Use Statement: The review does not specify which figure panels contain defects or provide verifiable evidence of generative AI image synthesis. We explicitly confirm that no generative AI image tools were used to create any figures in the manuscript; all diagrams and plots were produced via standard programmatic plotting and diagramming workflows, fully consistent with our AI Use Statement (
Section 7). -
Citation Correction: We have removed the duplicate reference entry and consolidated Make-a-Video into a single citation (
Singer et al., 2023) in the References section.
We hope that these clarifications can resolve your concerns. Given these substantial manuscript updates, we respectfully hope you might consider re-evaluating our paper and overall rating. Please let us know if any further questions remain.
Thanks to the authors for the rebuttal and the manuscript revisions. While several clarifications are helpful, most of my major concerns remain unresolved:
- Re-framing the multi-agent DAG through qualitative design intuitions (context isolation, parallel execution, and deferred synthesis) does not substitute for a controlled empirical baseline. Since the evaluation pipeline is the core methodological contribution, demonstrating its advantage over a single strong LLM judge in terms of alignment, bottleneck and cost remains essential.
- The newly added duration statistics (Appendix I) show that only 28 of the 80 metadata entries explicitly require durations ≥ 60s, with no length-conditional performance breakdown or completion rates provided. Moreover, the human validation is restricted solely to Dreamina outputs while still omitting key protocol details (e.g., annotator expertise, the number of evaluated pairs, and the exact ICC formulation).
- More concerningly, instead of providing the requested completion rates and length-conditional performance breakdowns, both the rebuttal and Appendix I insert a defensive disclaimer stating that such analyses "require output-level correspondence to the specified metadata entries and a declared completion rule" and cannot be inferred from the aggregate summaries. These reads like an unverified LLM-generated disclaimer rather than a substantive answer (similar to Table 6 stating that "absolute activation counts are not reconstructed from these rounded percentages").
- I am not comfortable with the insisting claim that all diagrams were produced solely via standard programmatic plotting and diagramming workflows. Figure 1 clearly exhibits raster-level visual artifacts (e.g., typical generative noise and inconsistent background colors across icon boxes), cropping defects (e.g., the icons in "Hierarchical Rubric & Diagnosis Report"), and distorted icon details (e.g., the icons for "Personalized Prompt", "Profile-Aware Overall Score", and "Per-Checkpoint Scores (0-1)") that are characteristic of generative image models rather than vector graphics software, alongside text defects (e.g., "Muti-Agent Evaluation" in Panel 3). Simply denying the use of generative tools without replacing or fixing the figure does not resolve my concern regarding manuscript verification and consistency with the AI Use Statement.
Overall, I find the work rather rough in execution and insufficiently verified by the authors, and maintain significant reservations about its methodological soundness. In its current form, the paper is unsuitable for publication at ICLR. I encourage the authors to substantially revise the manuscript for a future submission.
Rebuttal Response (1/2)
We thank the reviewer for the feedback and would like to clarify several aspects of our primary contributions in response to the reviewer’s concerns. Below, we address each of the points with corresponding manuscript updates.
1. W1 & Q1: Benchmark Contribution and Multi-Agent DAG Design Rationale
DirectorBench's primary contribution is an executable diagnostic benchmark protocol—integrating structured task specifications, decoupled user profiles, and a 40-checkpoint rubric—whose multi-agent DAG is specifically designed for dimension-relevant context isolation, concurrent execution, and deferred cross-modal synthesis.
Benchmark Protocol Positioning (
Sections 1 & 5): Rather than claiming standalone pipeline modules as isolated algorithmic inventions, DirectorBench establishes an executable diagnostic benchmark protocol for long-form agentic video generation. Its measurement validity is directly verified against human perceptual judgments (Table 1: 0.739 case-level Spearman ρ, 84.3% pairwise ranking accuracy, 96.5% top-3 bottleneck coverage) and repeated evaluations (α=0.828, activation Jaccard =0.994).Architectural Intuition for Multi-Agent Decomposition (
Section 2.4): Decomposing evaluation into a multi-agent DAG provides three concrete functional advantages over a single monolithic prompt:Dimension-Relevant Context Isolation: Each specialist agent (Script, Video, Audio, Stability) receives only the rubric instructions, task fields, and tool traces relevant to its dimension. This prevents irrelevant context from saturating the prompt and avoids premature cross-dimensional interference (e.g., visual quality biasing script scoring).
Parallel Execution: Independent specialist agents execute concurrently, reducing the sequential wall-clock latency of serial multi-dimension evaluation.
Deferred Joint Synthesis: Cross-modal judgments (text-video, video-audio, text-audio) and final synthesis execute only after independent specialist assessments are available, giving local scoring, relational checks, and bottleneck synthesis distinct responsibilities.
Section 2.4.2. W2 & Q2: Evaluator Bias and Cross-Judge Backbone Validation (GPT-4o vs. GPT-5.4)
Re-evaluating outputs with GPT-4o alongside GPT-5.4 yields 92.1% checkpoint-level agreement (κ=0.84), sample-level Spearman ρ=0.93, and an unchanged ranking of the evaluated models (Kendall's τ=1.00).
Section 6): To assess sensitivity to the choice of judge backbone, we re-evaluated the outputs using GPT-4o as a second judge backbone:[0.40, 0.60]Observed Consistency: The comparison yields high checkpoint agreement and preserves the ranking of the evaluated models under this backbone substitution.
Manuscript Revision: Added the "Sensitivity to Judge Backbone" analysis in
Section 6.3. W3 & Q3: Metadata-First Activation vs. Released Code Snapshot
All reported experiments in the paper were produced with metadata-first activation enabled (
Section 2.3,Section 3.2,Appendix B,Table 5).Execution Configuration: In all reported experiments, checkpoints required by the task metadata remain active even when the requested content is omitted and receive a normalized score of 0, whereas genuinely inapplicable optional checks are marked N/A.
Snapshot Boundary Clarification: As documented in
Appendix B, output-only routing is strictly a boundary of the released snapshot filter rather than the experimental configuration used to generate the paper's results.Manuscript Revision: Explicitly stated the experimental configuration in
Section 2.3andSection 3.2, and formalized the policy inAppendix B(Table 5).