acceptodds
Withdrawn from ICLR 2027

AgentDirector: A Movie Recap Benchmark for Long-Horizon Multimodal Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing multimodal benchmarks probe perception, reasoning, and generation in isolation, yet provide limited diagnostic insight into how these capabilities remain coordinated across long sequences of planning, tool use, and verification. We introduce AgentDirector, a 70-task benchmark that evaluates this capability through movie recap production from full-length films. The workflow couples narrative design and footage localization with narration synthesis, audiovisual editing, and inspection of intermediate results. We decompose this workflow into five settings spanning Planning, Execution, and End-to-End Production, varying supplied narration and plans, temporal requirements, and deliverables to examine these capabilities and their coordination. Our evaluation decomposes performance into Completeness, Verification, and Quality, with criteria aligned to each setting's responsibilities. These complementary components distinguish delivery failures, constraint violations, and weaknesses in narrative or audiovisual quality. The benchmark further supports systematic comparisons across agents and harnesses in task performance, editing quality, and computational cost. Component scores and execution traces help identify capability profiles and failure patterns throughout recap production.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agents, Long-Horizon Tasks, Agent Benchmarking, Movie Recap Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xNVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgentDirector benchmarks movie recap production from full-length films. Its 70 tasks cover five settings that separate planning, execution, and end-to-end production. Evaluation combines artifact completeness, constraint verification, and content and audiovisual quality. Experiments with eight model–harness configurations show that delivering a complete video does not necessarily imply accurate narration or well-grounded footage.

Critical strengths

The task decomposition provides a useful way to study failures that ordinary video-understanding benchmarks miss. Separating delivery, factual content, and footage support reveals that technically complete outputs can still be poor recaps. This makes the benchmark more informative than a single end-to-end leaderboard.

Critical weaknesses
  1. The human study supports the overall ordering of video preferences, but does not validate the individual Content and Sourcing judgments behind the main conclusions. This matters because the content judge partly relies on its own knowledge of the film, and “unverifiable” decisions are excluded. A human audit is needed to distinguish agent errors from evaluator errors and to check whether judgment coverage differs across systems.
  2. From-scratch production permits a coherent narrative different from the reference recap. However, localization rewards coverage of a fixed reference event set and overlap with reference intervals. It is unclear how the metric distinguishes unsupported footage from a reasonable alternative choice of events or shots. This affects the interpretation of Sourcing scores for open-ended production.
Questions for authors
  1. Can you report human agreement and error analysis for a representative sample of Content decisions, event matches, and footage–narration judgments, including the frequency of “unverifiable” decisions?
  2. Which reference events are essential for every valid recap, and how are alternative narratives handled? Would auditing valid alternative edits change the localization scores or system rankings?
Reviewer_43QnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentDirector, a benchmark of 70 tasks in which multimodal agents produce Chinese recap videos from entire films, testing whether they can keep understanding, planning, footage localization, editing, and verification of intermediate results coordinated across a long workflow. The tasks fall into planning, execution, and full production, with five settings that vary the supplied narration, plans, and timestamp requirements to separate responsibilities. Scoring multiplies completeness, verification, and quality (content accuracy, sourcing, and production), so different kinds of failure can be told apart. Experiments on eight model and harness configurations show that Claude Opus 5 leads overall but every model struggles with fully autonomous production from scratch; high completeness and compliance coexist with weak narrative accuracy and footage support; similar scores hide large runtime differences; and composite rankings disagree with human and LLM preferences for the finished videos.

Critical strengths

Strength 1 The five settings share one recap production workflow and differ only in whether narration, plans, and source timestamps are supplied, which separates planning, execution, and overall coordination as distinct responsibilities. This lets the results reveal patterns that a single aggregate score would hide, such as GLM-5.3 planning better than Omni-Flash yet executing much worse, and every model dropping sharply in production from scratch. The task is also realistically long and demanding: entire films, hours of runtime, and hundreds of tool calls per run.

Strength 2 Scoring multiplies delivery completeness, verifiable constraints, and quality, and splits quality into content accuracy, footage sourcing, and production, with deterministic checks and model judgments handling separate concerns. This decomposition localizes failure types: Opus 5 scores above 0.85 on completeness and compliance, yet content accuracy sits around 0.4 when it writes the story itself, and sourcing stays between 0.49 and 0.58 even with narration supplied. The protocol is further complemented by efficiency analysis and a preference study combining pairwise LLM judgments with human rankings, whose Spearman correlation of 0.976 lends credibility to the automatic scores.

Critical weaknesses

Weakness 1 The 70 tasks come from only 14 films, the planning and from scratch settings have just 10 instances each, and the film pools differ across settings. Every table reports point estimates with no confidence intervals or significance tests, yet the text draws conclusions from gaps of a few hundredths, such as 0.351 versus 0.344 between GPT-5.6 Sol and GLM-5.3 in production from scratch. The eight configurations also run under different APIs, thinking modes, context and output limits, and even different harnesses (GPT-5.6 Sol runs on Codex), while Appendix A.1 itself shows that swapping the harness alone can move scores by 0.13, so the main leaderboard is not a controlled comparison; the authors concede this in the appendix, but the claims in the abstract and main text are not softened accordingly.

Weakness 2 Content accuracy, footage and narration consistency, production quality, pairwise preference, and even the event matching that precedes temporal scoring all come from one judge, gemini-3.8-flash, with no reported agreement with humans on individual judgments and no sensitivity analysis using other judges. The only human study covers the overall ranking of 8 models in a single setting; the authors note that it does not measure agreement on individual videos or judgments, and a correlation of 0.976 over 8 points is thin evidence of judge reliability. The content judge is also allowed to use its own knowledge of the film, so reliability likely varies with how well known each film is; together with unjustified fixed weights (0.5/0.3/0.2 and the eight content category weights) and the paper’s own finding that composite rankings diverge from preferences for the finished videos (Qwen3.8-Max is seventh by score, second by LLM pairwise preference, and first by mean human rank), it is unclear what the composite score actually measures.

Questions for authors

See Weaknesses

Reviewer_BVtPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentDirector, a 70-task movie recap benchmark covering planning, execution, and end-to-end production. Its evaluation separates completeness, verification, and quality. Experiments compare eight model-harness configurations and analyze performance, resource use, and human preferences. The benchmark addresses a meaningful problem, although broader coverage and stronger validation of its diagnostic scores would strengthen the paper.

Critical strengths
  1. Meaningful and realistic tasks. Movie recap production requires agents to coordinate understanding, planning, footage retrieval, and audiovisual editing throughout long workflows.
  2. Useful evaluation design. The five task settings and the separation of completeness, verification, and quality help distinguish successful delivery from a correct and well-produced recap.
  3. Informative experimental analysis. The paper evaluates several model-harness configurations and includes resource analysis, a limited same-model harness comparison, and human rankings of completed videos.
Critical weaknesses
  1. Limited benchmark coverage. The 70 tasks derive from only 14 films, and all require Chinese narration. Repeated runs do not increase independent film coverage. Uncertainty estimates that account for film reuse would also help assess the reliability of the rankings.
  2. Insufficient validation of component scores. The reported human agreement concerns aggregate model rankings for completed videos, rather than individual content and sourcing judgments. Component-level human validation and sensitivity analyses for score weights and the exclusion of unverifiable items would strengthen the evaluation.
  3. Limited evaluation of specialized video-editing agents. The experiments mainly evaluate general-purpose model-harness configurations. Comparisons with specialized systems, such as VideoAgent, GLANCE, or DIRECT, with task-compatible adaptations and comparable computational budgets, would strengthen the benchmark evaluation.
Questions for authors
  1. How do specialized video-editing systems, such as VideoAgent, GLANCE, or DIRECT, perform on the compatible AgentDirector settings under comparable computational budgets? Please clarify any required adaptations and which task requirements prevent direct comparison.
  2. Can you validate individual content and sourcing judgments against human annotations, beyond the reported agreement on aggregate video-preference rankings? Please also report ranking uncertainty using the film, rather than each reused task or trial, as the clustering unit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.