- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TimelineBench, a benchmark for computer-use agents that perform nonlinear video editing. The target output is an editable project that a human editor can continue to revise. Each task provides a prepared project and a natural-language instruction. A rule-based judge then checks three things: whether the requested edits are present, whether the content that should stay unchanged was preserved, and whether the project contains any other unauthorized change. The agent sees screenshots and GUI actions, not the internal project or the evaluation rules.
Tasks are built from 45 official tutorials for Premiere Pro, Final Cut Pro, and DaVinci Resolve. The authors extract 165 workflows, organize them into 48 atomic editing operations and 121 reusable templates, and bind those templates to different media, targets, and parameter values. The resulting benchmark has 781 tasks. Instructions are first compiled from the task specification, then rewritten and checked by GPT-6 Astra. The main test environment is OpenCut-TL, a fork of the open-source OpenCut editor in which the authors added 19 operations. A 56-task subset is also run in Adobe Premiere Pro. Human annotators rate 200 generated instructions, and ten novice participants attempt 100 tasks.
Seven agents are each run once on all 781 OpenCut-TL tasks. Success ranges from 3.46% for OpenCUA-7B to 87.20% for GPT-6 Sol. On the 100-task human subset, novices score 64.5% and GPT-6 Sol scores 92%. On the matched 56 tasks, GPT-6 Sol falls from 83.93% in OpenCut-TL to 62.50% in Premiere Pro.
Critical strengths
-
TimelineBench tests computer-use agents on video-editing instructions. The agent works through the GUI. Scoring uses the saved project: the requested edit must be present, protected content must be unchanged, and no extra change is allowed.
-
The 781 tasks come from official Premiere Pro, Final Cut Pro, and DaVinci Resolve tutorials, organized as 48 edit o-perations and 121 templates. The full run is in OpenCut-TL, a modified version of OpenCut. A 56-task subset is repeated in Premiere Pro. Annotators rate 200 instructions, and ten novices attempt 100 tasks.
-
Seven agents are tested once each. Success on OpenCut-TL ranges from 3.5% to 87.2%. On the novice subset, GPT-6 Sol scores 92%, compared with 64.5% for the novices. On the Premiere subset, Sol drops from 83.9% to 62.5%.
Critical weaknesses
-
The aggregate success rates do not support the paper’s claims about professional video-editing ability. Of the 781 tasks, 303 contain only one atomic operation, and the six hardest length groups together contain 34 tasks. Fifty of the 121 templates are text and graphics, many of them recombinations of font, size, color, alignment, and position. GPT-6 Sol misses 1.1% of text conditions, compared with 16.9% for timeline editing and 23.2% for animation, and the animation category has only 56 tasks. The primary environment is also a simplified editor: the authors added 19 of the 48 operations, and several are exposed through property-panel controls. The only external check drops GPT-6 Sol from 83.93% to 62.50% on 56 Premiere tasks whose sampling procedure is not stated. The human baseline consists of graduate students with little or no editing experience, and skipped assignments count as failures. The abstract says preservation failures are a major source of error, but 89 of GPT-6 Sol’s 100 failures are unmet requests, and its preservation coverage is 99.82%. The pooled average is a poor tracker of progress on video-editing agents.
-
The automatic judge is assumed correct. The human study asks whether instructions look meaningful, self-contained, and fluent. It does not test whether the judge accepts professionally valid projects and rejects invalid ones, including an alternative effect stack, a different keyframe realization, or an extra but reasonable change. The reference program is built from the same specification as the evaluation rules, so agreement with it does not validate the judge. The same model, GPT-6 Astra, both rewrites the instructions and audits those rewrites. Premiere identifies objects by track, media, and timing because object IDs are unstable, and the paper reports no agreement rate between the two project parsers. The claim that a rendered video is an inadequate observation is also unmeasured: there is no count of episodes on which a visual judge and the project-state judge disagree. Runs are single-shot, and the open-weight agents receive a weaker observation history and a smaller output limit than the GPT agents. The ranking is not yet a validated measurement.
Questions for authors
-
Please report success and both coverage metrics so that each operation family, or each template, contributes equally, and report the same metrics on timeline-editing tasks and animation tasks alone. If those figures move substantially relative to 87.20%, the pooled success rate should not be the lead result.
-
On a sample of final projects stratified by success, unmet requests, and preservation or unauthorized-change failures, what fraction of judge decisions does an experienced editor agree with? Please include editorially reasonable alternative projects. On paired projects, how often do the OpenCut exporter and the Premiere exporter disagree?
-
How was the 56-task Premiere subset selected? Were tasks that annotators marked as not self-contained removed from the 781, and how were the 200 annotated tasks sampled? The positive rate for self-containedness is 92.8%.
-
Does the ranking persist if UI-TARS and OpenCUA receive the same history, output limit, and action budget as the GPT agents, with repeated trials on a shared subset?
9 comments
Official Rebuttal [2/7]
W1-2. The primary environment is also a simplified editor: the authors added 19 of the 48 operations, and several are exposed through property-panel controls.
Thank you for your comment. We use OpenCut-TL for scalable evaluation and Premiere for evaluation through a professional editing interface. The 19 added operations broaden OpenCut’s editing capabilities, while some property-panel controls make precise edits more direct than the interaction paths agents used in Premiere.
We report the two environments separately to capture both scalable task evaluation and performance through a professional interface. We have clarified this distinction in Section 6.5. The matched Premiere results, interaction examples, and subset selection criteria are discussed in Official Rebuttal [4/7] and Appendix D.5.
W1-4. The human baseline consists of graduate students with little or no editing experience, and skipped assignments count as failures.
Thank you for raising this concern. We added an experienced-participant baseline on the same 100 tasks, extending the human comparison beyond novice users. Five participants with video-editing experience performed one assignment per task.
| Participant / model | SR (%) |
|---|---|
| Human (novice) | 64.50 |
| Human (editing experience) | 71.00 |
| GPT-6 Sol | 92.00 |
| GPT-6 Luna | 76.00 |
| UI-TARS-1.5-7B | 6.00 |
| OpenCUA-7B | 9.00 |
The experienced group achieved 71.00% success, with no skipped assignments. For comparison, novice success was 64.50% with skips counted as failures and 69.35% among completed executions. Skipped novice assignments receive zero request and preservation coverage. We have added the experienced-participant baseline and clarified skip handling in Section 6.6 (Table 3) and Appendix D.6 (Table 24).
W1-5. The abstract says preservation failures are a major source of error, but 89 of GPT-6 Sol’s 100 failures are unmet requests, and its preservation coverage is 99.82%.
Thank you for your feedback. We revised the abstract’s emphasis to reflect the observed failure composition. On the expanded 1,453-task benchmark, 138 of GPT-6 Sol’s 154 failed episodes (89.6%) are request-only failures, with no preservation violations. Unmet requests therefore account for most of its failures.
The revised abstract emphasizes uneven performance across editing categories and partial fulfillment revealed by coverage metrics. Section 6.4 (Figure 3) distinguishes request-only failures from preservation and other failure types.
Official Rebuttal [3/7]
W2-1. The automatic judge is assumed correct. The human study asks whether instructions look meaningful, self-contained, and fluent. It does not test whether the judge accepts professionally valid projects and rejects invalid ones, including an alternative effect stack, a different keyframe realization, or an extra but reasonable change. The reference program is built from the same specification as the evaluation rules, so agreement with it does not validate the judge. The same model, GPT-6 Astra, both rewrites the instructions and audits those rewrites.
Q2-1. On a sample of final projects stratified by success, unmet requests, and preservation or unauthorized-change failures, what fraction of judge decisions does an experienced editor agree with? Please include editorially reasonable alternative projects.
Thank you for requesting this validation. We conducted an additional human evaluation of 200 final projects with five participants with video-editing experience, followed by a review of disagreements with the evaluator.
Human validation. Each project received two ratings. We reused the 200 tasks from the task-quality annotation study, whose sampling procedure appears in Appendix A.6, but asked participants to assess the resulting projects rather than instruction quality. We selected one recorded model episode per task, comprising 100 evaluator successes, 60 request-only failures, 20 preservation or unauthorized-change failures without unmet requests, and 20 failures involving both.
Agreement and disagreement analysis. Blinded human judgments agreed with the evaluator in 80.1% of determinate responses (318/397). We then examined all 79 disagreements against the instructions and before-and-after project states. Our author-led review identified 62 responses in which human raters overlooked explicit instruction violations that the evaluator detected. Examples included accepting an unchanged clip despite a replacement request, accepting bold text despite a request for normal weight, and overlooking unrequested changes to title styling or clip opacity.
Applying these author-led classifications to the ratings yields 95.7% agreement (380/397). The remaining 17 responses included differences over evaluation criteria, such as whether a one-frame timing deviation within the predefined tolerance should count as a failure. The initial 80.1% measures blinded human–evaluator agreement, while 95.7% summarizes the subsequent author-led analysis. These cases illustrate how explicit project-state checks can identify violations missed during human review. We have added the study design, agreement results, and disagreement analysis to Appendix C.2.
Reference validation and alternative solutions. The reference program provides one executable solution for validating task construction. The evaluator checks task requirements rather than matching the reference project. For example, B-roll can occupy two overlay tracks instead of the reference’s single track when track placement is unspecified, provided the required source ranges, timing, narration, and audio settings are satisfied. This allows alternative valid project structures. All 1,453 tasks passed reference-based validation, which we describe as a specification–reference–evaluator consistency check in Section 4.2 and Appendix A.5.
Instruction verification. GPT-6 Astra’s author and critic roles revise and check instruction wording against a fixed task specification. As reported in Section 4.3, human annotators evaluated the resulting instructions for meaningfulness, self-containedness, and language quality. We also conducted an additional assessment of practical editing purposes with participants with video-editing experience.
Official Rebuttal [1/7]
We thank the reviewer for highlighting our saved-project evaluation of requested edits, protected content, and unauthorized changes, together with the tutorial-grounded task construction.
Thank you for requesting this breakdown. We added success, request coverage, and preservation coverage under equal category and template weighting, together with separate timeline-editing and animation results. The macro comparisons preserve the top five models’ ordering, while category-specific results reveal differences in editing capability.
Dataset expansion. We expanded the dataset from 781 to 1,453 tasks to improve coverage of underrepresented editing categories and longer compositions. Each category now includes at least 100 distinct task bindings. Animation-containing tasks increased from 56 to 155, and tasks requiring at least five atomic operations increased from 34 to 130.
Equal-weight aggregation. Macro metrics average task-level values within groups, then weight group means equally. Table 1 compares these with pooled results on the same 1,453 tasks.
Table 1. Aggregation comparison (1,453 tasks). Each cell reports success / request coverage / preservation coverage (%).
The requested comparison shows that the top five models retain their ordering, but weighting affects their success rates differently. Category macro raises GPT-6 Luna and GPT-5.6 Luna by 4.63 and 4.89 percentage points, respectively. Template macro raises GPT-5.4 mini by 8.88 points and reverses the two open-weight models’ ordering. This comparison distinguishes stable model rankings from sensitivity to the task distribution. We have added the results and sensitivity analysis to Section 6.2 and Appendix D.2 (Table 16).
Category-specific performance.
Table 2. Timeline-editing and animation results. Each cell reports success / request coverage / preservation coverage (%).
The requested category-specific breakdown reveals lower success on animation than timeline-editing tasks across all evaluated models. Coverage adds another distinction: GPT-6 Luna achieves 74.86% request coverage and 99.57% preservation coverage on animation tasks, but only 46.45% task success. Together, these results reveal differences across editing capabilities and partial fulfillment within unsuccessful tasks. We have incorporated category-level findings into Section 6.4, the partial-fulfillment analysis into Section 6.2, and detailed results and analysis into Appendix D.4.1 (Tables 19–20).
We have updated the main results for the 1,453-task benchmark (Section 6.2, Table 1), added binding criteria and dataset distributions (Section 4.2, Appendix A.5, Tables 8–9), and defined the macro metrics (Section 5.3, Appendix C.1). Task-level analysis and editing examples appear in Appendices D.4.2 (Table 21) and D.4.4.