- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VAHBench asks whether an agent can turn a short-drama brief into a finished multi-episode video. Five English briefs specify story, cast, dialogue, format, and duration (22 episodes and 69 minutes per configuration). Twelve systems — six general-purpose harnesses, three of them also run with the OpenMontage tool library, and three purpose-built video agents — are each paired with four language models and shared image and video generators, giving 240 single runs. The evaluator is designed to score genuine completeness , basic quality (specification, consistency, plausibility), and a process score (reliability and honesty). Fixed rules assign the numbers; a language model only parses logs. The reported results concern and : Claude Code has the highest mean total (), Codex has higher but much lower , completeness falls from B1 to B5, and OpenMontage raises completeness while lowering quality, helping Codex and OpenClaw but hurting Claude Code. Process scores are not part of the reported analysis.
Critical strengths
The completeness definition targets a real failure mode of long-form agentic video. Equation (1) and Table 6 show that padding, looping, or a surviving episode cannot stand in for requested genuine duration: the same 40 seconds, padded from 40s to 59s, raises a container-duration baseline from 0.667 to 0.983 while lowering from 0.417 to 0.316. This is the right quantity for this task, and the analytical controls make the rule auditable.
The comparison is also broad enough to separate delivery from clip-level scores. On 180 runs that share the same three text models, and induce opposite orderings on 55.7% of untied matched system pairs, and their Spearman correlation is about . The OpenMontage effect is specific rather than uniform: over 15 matched pairs, mean changes by about +14.4 for Codex, +14.9 for OpenClaw, and for Claude Code, with completeness up and quality down in each case. If the quality measurements hold, that is a useful finding about tool access.
Critical weaknesses
-
is not checked against human labels. Appendix A.2 says the cut points still need calibration. Appendix A.3 notes that static or low-diversity video can score well on consistency. Codex looks like that case: consistency 0.939 at , which puts its (0.616) above Claude Code (0.513, , consistency 0.295). follows (Spearman 0.956), so the claimed split between quality and completeness hangs on an untested detector.
-
The experiment is narrower than the benchmark described in the paper. is part of , but Section 4.1 and Appendix F say it is missing for all 240 runs, so Table 2 is gated by . Each cell is one run, and the five briefs change length, aspect ratio, and story together. Appendix E also changes the setup: VideoClaw skips its confirmation step, and ViMax turns off portrait rejection and daily generation limits. Figure 6 is not taken from the scored runs (Appendix D.1). These rankings are too thin to rely on.
Questions for authors
-
On a sample that includes looped and still-image episodes, how well do genuine-duration, identity, contact, and subtitle checks match human labels? After that, does Codex still outrank Claude Code on quality?
-
Can be computed from the saved logs? If not, please confirm that every reported leaves it out.
-
What time, token, and generation budgets were used, and how often did they stop a run? Does a second seed keep the same order on and ? How much did the Appendix E changes move VideoClaw and ViMax?
-
Which lines in the briefs are actually scored? A large block is marked optional, and each brief asks the agent to consult the user, while Section 3.1 gives no human feedback.
2 comments
Response to Reviewer YQKo
Thank you again for your time and thoughtful feedback. As the discussion deadline is approaching, we would be grateful if you could take another look at our responses, if your schedule permits. Please let us know if any concerns remain or if further clarification would be helpful. We appreciate your time and consideration.
Response to Reviewer YQKo
As the discussion period draws to a close, we wanted to briefly follow up on the additional clarifications we posted yesterday. We would appreciate it if you could take a look when you have a chance and let us know whether they address your concerns or if any questions remain. Thank you for your time and consideration.