- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TIMELINE-BENCH, 56 professional video-editing tasks. Each task gives an agent raw footage, a brief, and a containerized toolchain, and asks for one finished video. The sources are purchased EditStock projects (11), public Cinestudy exercises (15), and commissioned UGC and commercial jobs (15 each). A run resolves a task only when the output passes four delivery tests, six content guards, the task’s brief checks, and a quality test. The quality test is a three-model panel, z-scored with frozen constants and compared with a professional reference; per-collection margins are set so the pass rate does not exceed the human win-or-tie rate. The calibration study has 43 editors and 2,582 assessable blind judgments. Sixteen agents (frontier models in OpenCode, Codex CLI, or Claude Code, plus an editorial-guidance condition and a DaVinci Resolve computer-use condition) each attempt every task once. The highest point estimate is 15/56 (26.8%); the mean is 14.0%. Editors prefer the reference in 83.5% of judgments. Of 771 unresolved runs, 562 fail only the quality test. The authors argue that agents meet explicit requirements and still miss craft: they inspect stills and transcripts, cut at 0.72× the reference rate, check renders for defects rather than story or pacing, and claim success in 93% of runs.
Critical strengths
-
The benchmark scores a finished edit, and it keeps specification compliance separate from viewing quality. That split is what makes the result informative. Agents already deliver a valid file and pass most explicit brief checks; editors still prefer the reference on polish, shot selection, graphics, and pacing. The same gap appears in evidence that does not depend on the quality threshold. Agents cut at 0.72× the reference rate and hold static shots longer, and only 5.5% of the problems they report after checking a render concern pacing, story, or shot choice, against 66% of editor notes. The preference study is carefully instrumented: three editors per pair, answers locked until both videos have played through with sound, pay independent of the answer, and the agent ranking by preference share stays correlated at ρ ≥ 0.96 when any editor is left out, despite single-judgment Krippendorff’s α = 0.10.
-
The release is also built to be re-scored. Reference edits stay out of the container, and all 896 traces are audited for retrieval. Delivery tests are exercised with a mechanical oracle and with single-defect controls. Margins for the reported table are refit leave-one-agent-out, and constants for new submissions are frozen. The paper states the limit that matters: the quality test tracks editors per agent (ρ = 0.93) and does not reliably score one edit (κ = 0.15; per-task ρ = 0.32).
Critical weaknesses
-
Task resolution is fit to human win rates, not to human decisions on individual videos. For each collection, the margin is chosen so the pass rate stays close to, and does not exceed, that collection’s win-or-tie rate. On single edits, the panel agrees with the human majority only at κ = 0.15, about the same as one editor against the others (κ = 0.14). Two of the three judges never watch the video with sound. They see one frame per second, plus cut rate, longest static hold, loudness, and silence fraction. These are the same signals the paper later uses as evidence of weak craft. The released scoring notes say the UGC margin is 1.54 standard deviations, against 0.21–0.52 for the other collections, because judges prefer polished agent cuts while editors prefer the deliberately raw UGC references. Raising the threshold brings the pass rate down. It does not correct that disagreement. GPT-6 Astra is both a leading agent and one of the judges.
-
The experiments cannot separate the top of the leaderboard, and the large share of quality-only failures is partly built into the checks. With one run on 56 tasks, the intervals are about ±11 points. In Table 9, curated guidance versus the same Astra setup is 15 against 12 resolved tasks, with an adjusted p of 1. Several content limits sit just above the worst reference edit (frozen share 0.60 versus 0.56, dead air 5.0 s versus 4.8 s), so a poor edit can still pass every content check. The fact that 562 of 771 unresolved runs fail only the quality test does not show that compliance is solved.
Questions for authors
-
Please report how the quality-test decision lines up with the human majority (pass/fail against prefer-agent, tie, or prefer-reference), overall and by collection. Of the 125 resolved runs, how many does the human majority reject? Of the edits editors win or tie, how many does the test fail?
-
Please rescore the panel after removing the four file measurements from the GPT-6 Astra and Claude Opus 5.5 prompts. Does the ranking hold, especially on UGC? Please also report agreement with editors for Gemini 3.8 Flash alone, the only judge that watches the video with audio, and check whether a judge scores its own model family more leniently.
-
Cinestudy is public and has the highest editor win-or-tie rate (28.7%). The trace audit shows that agents did not download finished edits during the run. Was any check done for memorization of those edits from pretraining?
Thank you for the detailed response and additional experiments. The validation on held-out tasks and annotators, repeated runs, additional system comparisons, and revised claims about guidance substantively address several of my concerns. I also acknowledge that the additional human evaluations strengthen the evidence for a substantial quality gap between the evaluated systems and professional edits. However, I retain the following concerns about the reliability of the proposed reusable benchmark.
The implications of evaluator drift for comparability over time remain insufficiently resolved.
In the prospective evaluation, the frozen September constants predict a pass rate of 35.7%, whereas in-session scoring predicts 28.6%, a difference of 7.1 percentage points. The response attributes this discrepancy to drift in a hosted judge. This difference exceeds the reported range of run-to-run standard deviations of 1.3–3.7 percentage points and is therefore material to future system comparisons.
Scoring new submissions alongside the 919 human-labelled videos is a useful mitigation. However, the response does not yet demonstrate that this calibration procedure consistently recovers comparable rates and system comparisons across scoring dates. A clearly specified versioning and calibration protocol, together with repeated scoring of fixed outputs across sessions, would help establish this. Otherwise, the comparability of scores reported at different times remains uncertain.
Matching an individual editor does not fully validate the hard pass-rate metric.
The additional results reasonably establish that disagreement is not unique to the automated evaluator. Nevertheless, the test still exhibits false-fail rates of approximately 64–70%, with wide confidence intervals, on the conditional subsets defined in the response. These rates should not be interpreted as dataset-wide error rates, but they indicate substantial uncertainty in hard decisions for edits accepted by both reference editors.
I acknowledge that the primary estimand is the per-agent rate and that agent-level correlations are high. However, ranking correlation alone does not establish accurate calibration of absolute pass rates or reliable comparisons between closely performing systems. The addition of pass probabilities and expected resolution rates is welcome, but their calibration on independent data and the propagation of judging and calibration uncertainty into system-level scores still require evidence.
The validity of audiovisual quality assessment remains only partially established.
The input ablations and injected-defect experiments are useful, and the deterministic checks address the tested synchronization and audio-continuity defects. However, these checks primarily validate specific technical failures; they do not directly establish sensitivity to sound design, pacing, or audiovisual coordination.
The relatively small change in aggregate correlation after removing audio demonstrates robustness of the current aggregate results to this input change. It does not, by itself, validate the measurement of audio quality. Since the response acknowledges the absence of separate human ratings for sound or synchronization, targeted human validation or a more explicit restriction of the quality dimensions covered would still be helpful.
Overall, the additional evidence strengthens the conclusion that the evaluated systems fall short of professional editing quality. It does not yet fully resolve my concerns about scoring stability, the interpretation of absolute pass rates, and evaluation coverage. Based on the current response, I tentatively maintain my recommendation to reject.