acceptodds
Under review as a conference paper at ICLR 2027

VAHBench:A Harness Benchmark for Agentic Long-Video Generation

Abstract

AI agents are commonly used for everyday information retrieval and coding. Weask whether they can also interpret human requests and turn them into complete videos. We introduce VAHBench, a benchmark that tests this ability through fiveshort-drama briefs specifying story, characters, dialogue and duration. Agentsplan and carry out production using shared generation tools. Evaluation combines video completeness and basic quality with checks of production reliability and reporting accuracy. Across 240 runs covering 48 configurations of general-purpose and video-specific agents, higher quality scores do not necessarily mean more complete delivery. Missing episodes, repeated footage and static filler reveal gaps between producing a video file and fulfilling the brief. Added composition tools help some agents but do not consistently improve performance. VAHBench mea-sures how well agents translate user intent into finished video content and helpsidentify where that process breaks down.

Primary area:
datasets and benchmarks
Keywords:
benchmark, long video generation, multimodal agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YQKoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VAHBench asks whether an agent can turn a short-drama brief into a finished multi-episode video. Five English briefs specify story, cast, dialogue, format, and duration (22 episodes and 69 minutes per configuration). Twelve systems — six general-purpose harnesses, three of them also run with the OpenMontage tool library, and three purpose-built video agents — are each paired with four language models and shared image and video generators, giving 240 single runs. The evaluator is designed to score genuine completeness CC, basic quality QQ (specification, consistency, plausibility), and a process score PP (reliability and honesty). Fixed rules assign the numbers; a language model only parses logs. The reported results concern CC and QQ: Claude Code has the highest mean total (S=47.86S = 47.86), Codex has higher QQ but much lower CC, completeness falls from B1 to B5, and OpenMontage raises completeness while lowering quality, helping Codex and OpenClaw but hurting Claude Code. Process scores are not part of the reported analysis.

Critical strengths

The completeness definition targets a real failure mode of long-form agentic video. Equation (1) and Table 6 show that padding, looping, or a surviving episode cannot stand in for requested genuine duration: the same 40 seconds, padded from 40s to 59s, raises a container-duration baseline from 0.667 to 0.983 while lowering CC from 0.417 to 0.316. This is the right quantity for this task, and the analytical controls make the rule auditable.

The comparison is also broad enough to separate delivery from clip-level scores. On 180 runs that share the same three text models, CC and QQ induce opposite orderings on 55.7% of untied matched system pairs, and their Spearman correlation is about −0.03-0.03. The OpenMontage effect is specific rather than uniform: over 15 matched pairs, mean SS changes by about +14.4 for Codex, +14.9 for OpenClaw, and −11.2-11.2 for Claude Code, with completeness up and quality down in each case. If the quality measurements hold, that is a useful finding about tool access.

Critical weaknesses
  1. QQ is not checked against human labels. Appendix A.2 says the cut points still need calibration. Appendix A.3 notes that static or low-diversity video can score well on consistency. Codex looks like that case: consistency 0.939 at C=0.136C = 0.136, which puts its QQ (0.616) above Claude Code (0.513, C=0.814C = 0.814, consistency 0.295). SS follows CC (Spearman 0.956), so the claimed split between quality and completeness hangs on an untested detector.

  2. The experiment is narrower than the benchmark described in the paper. PP is part of SS, but Section 4.1 and Appendix F say it is missing for all 240 runs, so Table 2 is CC gated by QQ. Each cell is one run, and the five briefs change length, aspect ratio, and story together. Appendix E also changes the setup: VideoClaw skips its confirmation step, and ViMax turns off portrait rejection and daily generation limits. Figure 6 is not taken from the scored runs (Appendix D.1). These rankings are too thin to rely on.

Questions for authors
  1. On a sample that includes looped and still-image episodes, how well do genuine-duration, identity, contact, and subtitle checks match human labels? After that, does Codex still outrank Claude Code on quality?

  2. Can PP be computed from the saved logs? If not, please confirm that every reported SS leaves it out.

  3. What time, token, and generation budgets were used, and how often did they stop a run? Does a second seed keep the same order on CC and QQ? How much did the Appendix E changes move VideoClaw and ViMax?

  4. Which lines in the briefs are actually scored? A large block is marked optional, and each brief asks the agent to consult the user, while Section 3.1 gives no human feedback.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YQKo

As the discussion period draws to a close, we wanted to briefly follow up on the additional clarifications we posted yesterday. We would appreciate it if you could take a look when you have a chance and let us know whether they address your concerns or if any questions remain. Thank you for your time and consideration.

AuthorsOpenReview ↗

Response to Reviewer YQKo

Thank you again for your time and thoughtful feedback. As the discussion deadline is approaching, we would be grateful if you could take another look at our responses, if your schedule permits. Please let us know if any concerns remain or if further clarification would be helpful. We appreciate your time and consideration.

Reviewer_FB32OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces VAHBnech, a benchmark for evaluating whether AI agents can turn a detailed user brief into a complete long-form video series. VAHBench focuses on end-to-end production, including planning, tool use, media generation, video assembly and final delivery.

VAHBench contains five English short-drama briefs. Th evaluation framework separates three dimensions. Completeness measures whether all requested episodes and genuine video content are delivered. Basic quality evlauates format and brief compliance, character and cut consistenc, physical plausibility and etc.

Overall, VAHBench provides an initial framework for evaluating complete agentic video-production workflows rather than isolated generated clips.

Critical strengths
  1. VAHBench evaluates general-purpose harnesses without requiring a predefined multi-agent workflow. Each agent can decide how to plan, generate and assemble the video.
  2. End-to-end evaluation. The benchmark evaluates whether an agent delivers a complete video series and evaluate completeness and basic quality separately.
Critical weaknesses
  1. No complete video-brief alignment evaluation. The benchmark checks measurable requirements such as duration, format, require lines and subtitles, but does not evaluate whether the generated video follows the requested plot or narrativce intent.
  2. Unvalidated automatic metrics: The video quality detectors and thresholds are not validated against human annotations, making it unclear whether the scores agree with human judgements.
Questions for authors
  1. Can you test the human alignment with the proposed objective evaluation metrics?
  2. Can you add more evaluation metrics on video-brief alignment?
  3. Can you show more zero-shot examples to test generalizability?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FB32

As the discussion period draws to a close, we wanted to briefly follow up on the additional clarifications we posted yesterday. We would appreciate it if you could take a look when you have a chance and let us know whether they address your concerns or if any questions remain. Thank you for your time and consideration.

Thank you for the thoughtful response. It addressed most of my concerns, though I would still encourage a human study to assess how well the results align with human preferences. I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer FB32

Thank you for your response. We welcome further discussion. As clarified in our response, we believe that subjective numerical scores are not sufficiently reliable and can only indicate the relative ranking of A and B. We therefore suggest evaluating results along two clearer dimensions: completeness and visual quality. In our assessment, the ranking based on visual quality is consistent with that produced by our metric. Specific examples will be available in the visual results accompanying our open-source release.

Reviewer_5EBuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper points out that existing benchmarks score video clip quality or supply their own pipelines rather than the agent's native delivery ability. It introduces VAHBench: five short-drama briefs specifying story, characters, dialogue, and duration, where agents plan and execute with shared generation tools and no benchmark-supplied helpers, workflow, or rubric. Evaluation combines video completeness and basic quality with trajectory-level checks of production reliability and reporting honesty, using an LLM only to parse logs while fixed rules assign scores and reported delivery is verified against saved files. Across 240 runs over 48 configurations of general-purpose harnesses and video-specific agents, higher quality does not imply complete delivery — missing episodes, repeated footage, and static filler are common, and composition tools help only some agents.

Critical strengths
  1. VAHBench targets a neglected axis, whether an agent actually delivers the requested series, not whether clips look good, and its completeness metric is deliberately hard to hack: credited genuine duration nets out held, blank, repeated, broken-splice and picture-free intervals, and over-delivery is penalised in the denominator so padding cannot raise the score. The protocol is defensibly constructed: an LLM is used only to parse logs into structured claims, fixed rules assign every score, and the agent's reported delivery is verified against saved files, with a separate honesty facet that penalises over-claims while allowing abstention.

  2. The evaluation is broad: 48 configurations and 240 runs span general-purpose harnesses (Claude Code, Codex, OpenClaw, Hermes, OpenSquilla, DeepSeek), purpose-built video agents (UniVA, ViMax, VideoClaw), four text models per system, and a separate OpenMontage tool-library track, all sharing the same generation endpoints.

Critical weaknesses
  1. Every configuration runs once per brief, and independent reruns may produce different videos and scores; no repeated runs or variance are reported. Mid-table orderings and the reported tool-library gains may not be separated from run-to-run noise.

  2. Per-run cost, wall-clock time and tool-call budgets are never reported or equalised across configurations, even though agents visibly trade budget against strategy. A system that spends far more tokens and calls may score higher for that reason alone.

Questions for authors
  1. How does the ranking change when compute or call budget is matched across systems?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 5EBu

As the discussion period draws to a close, we wanted to briefly follow up on the additional clarifications we posted yesterday. We would appreciate it if you could take a look when you have a chance and let us know whether they address your concerns or if any questions remain. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.