acceptodds
Under review as a conference paper at ICLR 2027

TIMELINE-BENCH: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut

Abstract

AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we introduce TIMELINE-BENCH, a benchmark of 56 real video-editing tasks, each asking an agent to turn raw production material into a finished video. Tasks range from selecting dialog takes and shaping interview footage into a story to cutting commercials from product shots, voiceovers and graphics. Every task provides a brief, source assets, a container and a set of tests. A task is resolved when the output passes every test. The tests check the delivery format, the content and the brief's explicit requirements, and include a quality test calibrated on 2,582 blind judgments by 43 video editors. We evaluate 16 agents that pair frontier models with coding-agent harnesses such as Codex, Claude Code and OpenCode. The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of the 56 tasks (26.8%), and the average agent resolves 14.0%. Human editors prefer the reference edit in 83.5% of judgments. Most unresolved runs (562 of 771) fail only the quality test: agents perceive footage through stills and transcripts and check their renders for defects, not craft. We release the tasks, verifier and per-run results at https://timelinebench.vercel.app.

Primary area:
datasets and benchmarks
Keywords:
video editing, AI agents, multimodal benchmarks, agent evaluation, human evaluation, editorial quality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nD8KOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TIMELINE-BENCH, 56 professional video-editing tasks. Each task gives an agent raw footage, a brief, and a containerized toolchain, and asks for one finished video. The sources are purchased EditStock projects (11), public Cinestudy exercises (15), and commissioned UGC and commercial jobs (15 each). A run resolves a task only when the output passes four delivery tests, six content guards, the task’s brief checks, and a quality test. The quality test is a three-model panel, z-scored with frozen constants and compared with a professional reference; per-collection margins are set so the pass rate does not exceed the human win-or-tie rate. The calibration study has 43 editors and 2,582 assessable blind judgments. Sixteen agents (frontier models in OpenCode, Codex CLI, or Claude Code, plus an editorial-guidance condition and a DaVinci Resolve computer-use condition) each attempt every task once. The highest point estimate is 15/56 (26.8%); the mean is 14.0%. Editors prefer the reference in 83.5% of judgments. Of 771 unresolved runs, 562 fail only the quality test. The authors argue that agents meet explicit requirements and still miss craft: they inspect stills and transcripts, cut at 0.72× the reference rate, check renders for defects rather than story or pacing, and claim success in 93% of runs.

Critical strengths
  • The benchmark scores a finished edit, and it keeps specification compliance separate from viewing quality. That split is what makes the result informative. Agents already deliver a valid file and pass most explicit brief checks; editors still prefer the reference on polish, shot selection, graphics, and pacing. The same gap appears in evidence that does not depend on the quality threshold. Agents cut at 0.72× the reference rate and hold static shots longer, and only 5.5% of the problems they report after checking a render concern pacing, story, or shot choice, against 66% of editor notes. The preference study is carefully instrumented: three editors per pair, answers locked until both videos have played through with sound, pay independent of the answer, and the agent ranking by preference share stays correlated at ρ ≥ 0.96 when any editor is left out, despite single-judgment Krippendorff’s α = 0.10.

  • The release is also built to be re-scored. Reference edits stay out of the container, and all 896 traces are audited for retrieval. Delivery tests are exercised with a mechanical oracle and with single-defect controls. Margins for the reported table are refit leave-one-agent-out, and constants for new submissions are frozen. The paper states the limit that matters: the quality test tracks editors per agent (ρ = 0.93) and does not reliably score one edit (κ = 0.15; per-task ρ = 0.32).

Critical weaknesses
  1. Task resolution is fit to human win rates, not to human decisions on individual videos. For each collection, the margin is chosen so the pass rate stays close to, and does not exceed, that collection’s win-or-tie rate. On single edits, the panel agrees with the human majority only at κ = 0.15, about the same as one editor against the others (κ = 0.14). Two of the three judges never watch the video with sound. They see one frame per second, plus cut rate, longest static hold, loudness, and silence fraction. These are the same signals the paper later uses as evidence of weak craft. The released scoring notes say the UGC margin is 1.54 standard deviations, against 0.21–0.52 for the other collections, because judges prefer polished agent cuts while editors prefer the deliberately raw UGC references. Raising the threshold brings the pass rate down. It does not correct that disagreement. GPT-6 Astra is both a leading agent and one of the judges.

  2. The experiments cannot separate the top of the leaderboard, and the large share of quality-only failures is partly built into the checks. With one run on 56 tasks, the intervals are about ±11 points. In Table 9, curated guidance versus the same Astra setup is 15 against 12 resolved tasks, with an adjusted p of 1. Several content limits sit just above the worst reference edit (frozen share 0.60 versus 0.56, dead air 5.0 s versus 4.8 s), so a poor edit can still pass every content check. The fact that 562 of 771 unresolved runs fail only the quality test does not show that compliance is solved.

Questions for authors
  • Please report how the quality-test decision lines up with the human majority (pass/fail against prefer-agent, tie, or prefer-reference), overall and by collection. Of the 125 resolved runs, how many does the human majority reject? Of the edits editors win or tie, how many does the test fail?

  • Please rescore the panel after removing the four file measurements from the GPT-6 Astra and Claude Opus 5.5 prompts. Does the ranking hold, especially on UGC? Please also report agreement with editors for Gemini 3.8 Flash alone, the only judge that watches the video with audio, and check whether a judge scores its own model family more leniently.

  • Cinestudy is public and has the highest editor win-or-tie rate (28.7%). The trace audit shows that agents did not download finished edits during the run. Was any check done for memorization of those edits from pretraining?

Reviewer_PGm8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TIMELINE-BENCH, a benchmark containing 56 realistic video-editing tasks. Agents receive raw production materials and client briefs and must deliver finished videos. The evaluation covers delivery specifications, content integrity, brief compliance, and editorial quality. The quality test is calibrated using 2,582 assessable blind judgments from 43 video editors.

The authors evaluate 16 agent configurations. The best system achieves a resolution rate of 26.8%, with an average of 14.0% across agents. Most unsuccessful runs pass the technical and brief-related tests but fail the quality test. The paper also analyzes weaknesses in shot selection, pacing, and self-review.

Critical strengths

S1. Realistic video-editing tasks. The benchmark covers complete video-production assignments across different scenarios. The tasks include source materials, client briefs, and verified evaluation requirements. The professional review and test validation improve the credibility of the task set.

S2. Detailed analysis of agent failures. The paper combines human feedback, editing statistics, and execution traces to examine agent behavior. The findings on pacing, shot selection, and self-review are supported by several types of evidence and help explain the low task resolution rates.

Critical weaknesses

W1. Limited validation of the quality test on unseen tasks. The quality test plays a central role in determining task success. Automated resolution rates correlate well with human preferences at the agent level (ρ = 0.93), but Appendix D.4 reports low agreement on individual edits (κ = 0.15) and a per-task correlation of 0.32.

Section 6 includes a threshold sensitivity analysis. Lowering the quality margins by 0.1 standard deviations produces 27 additional successful runs, with the overall agent ranking remaining largely stable (Spearman ρ = 0.96). The authors also report similarly low agreement among human editors.

These results provide useful context for interpreting the evaluator. Its performance on tasks excluded from calibration remains unclear. Given its role in the final resolution rate, this deserves further validation.

W2. The incremental contribution over existing benchmarks needs clearer explanation. AgenticVBench already includes video-production tasks and uses programmatic checks, expert-defined criteria, and some human evaluation. The authors discuss related benchmarks in Appendix N, Table 11, and show how editorial quality tests affect the results.

The specific value of preference-calibrated evaluation relative to existing expert-based criteria is less clear. It would be helpful to identify which additional capabilities or failure modes this evaluation captures.

Questions for authors

Q1. Generalization of the quality test. Section 6 shows that agent rankings remain largely stable under a small change in the quality margins. Given the limited individual-edit agreement reported in Appendix D.4, could the authors evaluate the quality test on held-out tasks, for example through leave-one-task-out calibration, and report its agreement with human judgments? Q2. Run-to-run variability. Each agent is evaluated only once per task, and the leading agents differ by relatively few successful tasks. Could the authors repeat a representative subset of tasks for the leading agents and report the run-to-run variability in resolution rates?

Q3. Professional acceptability and relative quality. The judges provide explicit professional acceptability judgments, but these are excluded from the final quality score. Could the authors compare these judgments with the reference-relative pass decisions and report how often the two criteria disagree?

Reviewer_NC1wOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TIMELINE-BENCH, an end-to-end benchmark comprising 56 realistic video-editing tasks. Agents must produce complete videos from raw footage and task briefs. Evaluation covers delivery specifications, content integrity, brief compliance, and editorial quality. The paper compares 16 agent configurations using programmatic verification, multimodal model judges, and 2,582 assessable blinded judgments from 43 video editors.

The best configuration achieves a task resolution rate of 26.8%. Most unsuccessful runs satisfy explicit delivery and content requirements but fail the quality assessment. The authors conclude that current agents primarily struggle with pacing, shot selection, sound design, and overall polish.

Critical strengths
  1. The research problem is practically relevant. Instead of evaluating isolated editing operations or successful file generation, the benchmark covers the workflow from raw footage to a finished video. Separating technical compliance from editorial quality helps expose the gap between producing a valid video and delivering a professionally edited result.

  2. The human evaluation is substantial, and experimental reporting is transparent. The paper includes a blinded study with video editors and reports confidence intervals, paired statistical tests, and comparisons across execution frameworks. Reference isolation, assignment consistency checks, and injected-defect controls also demonstrate attention to evaluation reliability.

Critical weaknesses
  1. Validation of the core quality evaluator does not adequately support individual task-success decisions. Appendix D.4 reports only κ = 0.15 agreement between automated judgments and human majority decisions on individual videos, and ρ = 0.32 correlation between automated and human task-level rates. The authors explicitly acknowledge that the evaluator is primarily valid at the aggregate agent level. Nevertheless, it determines individual task success and supports the classification of failures attributed exclusively to quality. Strong aggregate correlation does not establish the reliability of these per-video labels. Human judgments are also noisy, which further motivates explicitly modeling uncertainty in success decisions. Moreover, leave-one-agent-out calibration does not establish generalization to independent tasks and annotators.

  2. The judges’ inputs are insufficient for fully assessing audiovisual quality. According to Section 3.3, two of the three judges receive only one-frame-per-second contact sheets and a few numerical measurements, while still scoring dimensions such as pacing and sound. These inputs cannot adequately convey music–cut alignment, audiovisual synchronization, audio transitions, or motion continuity—precisely the editing capabilities emphasized by the paper. Without systematic ablations comparing full audiovisual inputs with static-frame inputs, it remains unclear whether the evaluation captures these qualities or predominantly relies on static visual cues.

  3. The baselines do not sufficiently separate agent capability from limitations of tools and perception interfaces. The evaluation predominantly covers general-purpose coding agents, without a strong specialized video-editing system or an agent equipped with native audiovisual perception. Although multiple execution frameworks are compared, these experiments cannot determine whether poor editing quality reflects limited editorial reasoning or the use of still frames, transcripts, and tool outputs to understand videos. The sole GUI configuration simultaneously changes the environment, tool permissions, and perception interface, and cannot directly hear audio. Consequently, the code-versus-GUI comparison does not isolate the interaction paradigm, and broader capability claims should be restricted to the evaluated system configurations.

  4. Single-run evaluation and the limited task set weaken conclusions about rankings and intervention benefits. Each configuration is evaluated only once per task, so the reported intervals exclude run-to-run variability. Curated guidance increases resolved tasks from 12 to 15, but improves the human win-or-tie rate by only 0.6 percentage points; Table 9 reports this difference as nonsignificant. These findings support a substantial gap between the evaluated systems and professional reference edits, but do not establish a stable best configuration or a reproducible benefit from guidance. Multiple independent runs and cross-model evaluations of the guidance intervention are needed.

Questions for authors
  1. Can the quality evaluator be validated on independent tasks and annotators, with per-video false-pass and false-fail rates and confidence intervals? Given the low individual-video agreement, how reliable is the classification of quality-only failures?
  2. Can the authors compare full audiovisual judging, static-frame-only judging, and the current mixed panel, particularly for agreement with human assessments of sound, pacing, and audiovisual synchronization?
  3. Can specialized editing systems or agents with native audiovisual perception be included, alongside code-versus-GUI comparisons with matched tool permissions, perception capabilities, and resource budgets?
  4. Can the main configurations be evaluated over multiple independent runs, and can curated guidance be tested across several models to establish the stability of rankings and improvements?
1 comment

Thank you for the detailed response and additional experiments. The validation on held-out tasks and annotators, repeated runs, additional system comparisons, and revised claims about guidance substantively address several of my concerns. I also acknowledge that the additional human evaluations strengthen the evidence for a substantial quality gap between the evaluated systems and professional edits. However, I retain the following concerns about the reliability of the proposed reusable benchmark.

The implications of evaluator drift for comparability over time remain insufficiently resolved.

In the prospective evaluation, the frozen September constants predict a pass rate of 35.7%, whereas in-session scoring predicts 28.6%, a difference of 7.1 percentage points. The response attributes this discrepancy to drift in a hosted judge. This difference exceeds the reported range of run-to-run standard deviations of 1.3–3.7 percentage points and is therefore material to future system comparisons.

Scoring new submissions alongside the 919 human-labelled videos is a useful mitigation. However, the response does not yet demonstrate that this calibration procedure consistently recovers comparable rates and system comparisons across scoring dates. A clearly specified versioning and calibration protocol, together with repeated scoring of fixed outputs across sessions, would help establish this. Otherwise, the comparability of scores reported at different times remains uncertain.

Matching an individual editor does not fully validate the hard pass-rate metric.

The additional results reasonably establish that disagreement is not unique to the automated evaluator. Nevertheless, the test still exhibits false-fail rates of approximately 64–70%, with wide confidence intervals, on the conditional subsets defined in the response. These rates should not be interpreted as dataset-wide error rates, but they indicate substantial uncertainty in hard decisions for edits accepted by both reference editors.

I acknowledge that the primary estimand is the per-agent rate and that agent-level correlations are high. However, ranking correlation alone does not establish accurate calibration of absolute pass rates or reliable comparisons between closely performing systems. The addition of pass probabilities and expected resolution rates is welcome, but their calibration on independent data and the propagation of judging and calibration uncertainty into system-level scores still require evidence.

The validity of audiovisual quality assessment remains only partially established.

The input ablations and injected-defect experiments are useful, and the deterministic checks address the tested synchronization and audio-continuity defects. However, these checks primarily validate specific technical failures; they do not directly establish sensitivity to sound design, pacing, or audiovisual coordination.

The relatively small change in aggregate correlation after removing audio demonstrates robustness of the current aggregate results to this input change. It does not, by itself, validate the measurement of audio quality. Since the response acknowledges the absence of separate human ratings for sound or synchronization, targeted human validation or a more explicit restriction of the quality dimensions covered would still be helpful.

Overall, the additional evidence strengthens the conclusion that the evaluated systems fall short of professional editing quality. It does not yet fully resolve my concerns about scoring stability, the interpretation of absolute pass rates, and evaluation coverage. Based on the current response, I tentatively maintain my recommendation to reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.