acceptodds
Under review as a conference paper at ICLR 2027

TimelineBench: Evaluating Video-Editing Agents on Editable Project States

Abstract

In professional video editing, an agent's output is not a rendered video but an editable project that human editors continue to work on. Thus, rigorous evaluation of video-editing agents requires both scalable construction of realistic editing tasks and systematic verification of such project states. We introduce TimelineBench, a benchmark of 781 video-editing tasks that connects task construction and outcome evaluation through a shared structured specification. TimelineBench extracts real workflows from official editing tutorials and organizes them into 48 atomic editing operations and 121 reusable task templates. Each atomic operation specifies both the edit to perform and evaluation rules for the requested changes and required preservation, allowing natural-language instructions and their evaluation to come from the same task specification. TimelineBench evaluates agents directly on the resulting project state, verifying request completion and state preservation, without requiring a reference GUI trajectory or a single gold project. We build OpenCut-TL to run the full benchmark and evaluate seven computer-use agents, whose task success rates range from 3.5% to 87.2%, with preservation failures remaining a major source of error. By unifying realistic task construction with systematic outcome evaluation, TimelineBench provides a scalable and reproducible foundation for evaluating video-editing agents.

Primary area:
datasets and benchmarks
Keywords:
computer-use agent, gui agent, video editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Kaz1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TimelineBench, a benchmark for computer-use agents that perform nonlinear video editing. The target output is an editable project that a human editor can continue to revise. Each task provides a prepared project and a natural-language instruction. A rule-based judge then checks three things: whether the requested edits are present, whether the content that should stay unchanged was preserved, and whether the project contains any other unauthorized change. The agent sees screenshots and GUI actions, not the internal project or the evaluation rules.

Tasks are built from 45 official tutorials for Premiere Pro, Final Cut Pro, and DaVinci Resolve. The authors extract 165 workflows, organize them into 48 atomic editing operations and 121 reusable templates, and bind those templates to different media, targets, and parameter values. The resulting benchmark has 781 tasks. Instructions are first compiled from the task specification, then rewritten and checked by GPT-6 Astra. The main test environment is OpenCut-TL, a fork of the open-source OpenCut editor in which the authors added 19 operations. A 56-task subset is also run in Adobe Premiere Pro. Human annotators rate 200 generated instructions, and ten novice participants attempt 100 tasks.

Seven agents are each run once on all 781 OpenCut-TL tasks. Success ranges from 3.46% for OpenCUA-7B to 87.20% for GPT-6 Sol. On the 100-task human subset, novices score 64.5% and GPT-6 Sol scores 92%. On the matched 56 tasks, GPT-6 Sol falls from 83.93% in OpenCut-TL to 62.50% in Premiere Pro.

Critical strengths
  • TimelineBench tests computer-use agents on video-editing instructions. The agent works through the GUI. Scoring uses the saved project: the requested edit must be present, protected content must be unchanged, and no extra change is allowed.

  • The 781 tasks come from official Premiere Pro, Final Cut Pro, and DaVinci Resolve tutorials, organized as 48 edit o-perations and 121 templates. The full run is in OpenCut-TL, a modified version of OpenCut. A 56-task subset is repeated in Premiere Pro. Annotators rate 200 instructions, and ten novices attempt 100 tasks.

  • Seven agents are tested once each. Success on OpenCut-TL ranges from 3.5% to 87.2%. On the novice subset, GPT-6 Sol scores 92%, compared with 64.5% for the novices. On the Premiere subset, Sol drops from 83.9% to 62.5%.

Critical weaknesses
  1. The aggregate success rates do not support the paper’s claims about professional video-editing ability. Of the 781 tasks, 303 contain only one atomic operation, and the six hardest length groups together contain 34 tasks. Fifty of the 121 templates are text and graphics, many of them recombinations of font, size, color, alignment, and position. GPT-6 Sol misses 1.1% of text conditions, compared with 16.9% for timeline editing and 23.2% for animation, and the animation category has only 56 tasks. The primary environment is also a simplified editor: the authors added 19 of the 48 operations, and several are exposed through property-panel controls. The only external check drops GPT-6 Sol from 83.93% to 62.50% on 56 Premiere tasks whose sampling procedure is not stated. The human baseline consists of graduate students with little or no editing experience, and skipped assignments count as failures. The abstract says preservation failures are a major source of error, but 89 of GPT-6 Sol’s 100 failures are unmet requests, and its preservation coverage is 99.82%. The pooled average is a poor tracker of progress on video-editing agents.

  2. The automatic judge is assumed correct. The human study asks whether instructions look meaningful, self-contained, and fluent. It does not test whether the judge accepts professionally valid projects and rejects invalid ones, including an alternative effect stack, a different keyframe realization, or an extra but reasonable change. The reference program is built from the same specification as the evaluation rules, so agreement with it does not validate the judge. The same model, GPT-6 Astra, both rewrites the instructions and audits those rewrites. Premiere identifies objects by track, media, and timing because object IDs are unstable, and the paper reports no agreement rate between the two project parsers. The claim that a rendered video is an inadequate observation is also unmeasured: there is no count of episodes on which a visual judge and the project-state judge disagree. Runs are single-shot, and the open-weight agents receive a weaker observation history and a smaller output limit than the GPT agents. The ranking is not yet a validated measurement.

Questions for authors
  1. Please report success and both coverage metrics so that each operation family, or each template, contributes equally, and report the same metrics on timeline-editing tasks and animation tasks alone. If those figures move substantially relative to 87.20%, the pooled success rate should not be the lead result.

  2. On a sample of final projects stratified by success, unmet requests, and preservation or unauthorized-change failures, what fraction of judge decisions does an experienced editor agree with? Please include editorially reasonable alternative projects. On paired projects, how often do the OpenCut exporter and the Premiere exporter disagree?

  3. How was the 56-task Premiere subset selected? Were tasks that annotators marked as not self-contained removed from the 781, and how were the 200 annotated tasks sampled? The positive rate for self-containedness is 92.8%.

  4. Does the ranking persist if UI-TARS and OpenCUA receive the same history, output limit, and action budget as the GPT agents, with repeated trials on a shared subset?

9 comments
AuthorsOpenReview ↗

Official Rebuttal [1/7]

We thank the reviewer for highlighting our saved-project evaluation of requested edits, protected content, and unauthorized changes, together with the tutorial-grounded task construction.


W1-1. The aggregate success rates do not support the paper’s claims about professional video-editing ability. Of the 781 tasks, 303 contain only one atomic operation, and the six hardest length groups together contain 34 tasks. Fifty of the 121 templates are text and graphics, many of them recombinations of font, size, color, alignment, and position. GPT-6 Sol misses 1.1% of text conditions, compared with 16.9% for timeline editing and 23.2% for animation, and the animation category has only 56 tasks.

W1-6. The pooled average is a poor tracker of progress on video-editing agents.

Q1. Please report success and both coverage metrics so that each operation family, or each template, contributes equally, and report the same metrics on timeline-editing tasks and animation tasks alone. If those figures move substantially relative to 87.20%, the pooled success rate should not be the lead result.

Thank you for requesting this breakdown. We added success, request coverage, and preservation coverage under equal category and template weighting, together with separate timeline-editing and animation results. The macro comparisons preserve the top five models’ ordering, while category-specific results reveal differences in editing capability.

Dataset expansion. We expanded the dataset from 781 to 1,453 tasks to improve coverage of underrepresented editing categories and longer compositions. Each category now includes at least 100 distinct task bindings. Animation-containing tasks increased from 56 to 155, and tasks requiring at least five atomic operations increased from 34 to 130.

Equal-weight aggregation. Macro metrics average task-level values within groups, then weight group means equally. Table 1 compares these with pooled results on the same 1,453 tasks.

Table 1. Aggregation comparison (1,453 tasks). Each cell reports success / request coverage / preservation coverage (%).

ModelPooledCategory macroTemplate macro
GPT-6 Sol89.40 / 93.89 / 99.7490.54 / 95.56 / 99.7488.48 / 92.26 / 99.77
GPT-5.6 Sol84.65 / 90.22 / 99.3884.39 / 90.45 / 99.2385.80 / 90.48 / 99.49
GPT-6 Luna75.29 / 82.53 / 99.1279.92 / 86.97 / 99.2274.42 / 80.80 / 99.26
GPT-5.6 Luna71.71 / 80.34 / 99.1776.60 / 85.20 / 99.4871.44 / 78.84 / 98.98
GPT-5.4 mini31.04 / 43.02 / 92.0027.20 / 37.75 / 90.0839.92 / 54.63 / 93.94
OpenCUA-7B3.44 / 14.13 / 86.413.73 / 13.00 / 86.393.85 / 17.45 / 86.77
UI-TARS-1.5-7B3.17 / 12.99 / 88.782.37 / 9.59 / 87.894.58 / 18.11 / 89.09

The requested comparison shows that the top five models retain their ordering, but weighting affects their success rates differently. Category macro raises GPT-6 Luna and GPT-5.6 Luna by 4.63 and 4.89 percentage points, respectively. Template macro raises GPT-5.4 mini by 8.88 points and reverses the two open-weight models’ ordering. This comparison distinguishes stable model rankings from sensitivity to the task distribution. We have added the results and sensitivity analysis to Section 6.2 and Appendix D.2 (Table 16).

Category-specific performance.

Table 2. Timeline-editing and animation results. Each cell reports success / request coverage / preservation coverage (%).

ModelTimeline-editing tasks (776)Animation tasks (155)
GPT-6 Sol83.12 / 90.44 / 99.5762.58 / 85.63 / 98.66
GPT-5.6 Sol79.12 / 87.99 / 99.2149.68 / 76.15 / 97.02
GPT-6 Luna64.18 / 75.86 / 99.0046.45 / 74.86 / 99.57
GPT-5.6 Luna57.47 / 71.82 / 98.7326.45 / 64.99 / 99.05
GPT-5.4 mini16.11 / 34.96 / 90.987.74 / 19.47 / 88.57
OpenCUA-7B0.90 / 11.66 / 86.830.00 / 4.68 / 85.50
UI-TARS-1.5-7B1.16 / 13.33 / 87.070.00 / 5.09 / 82.70

The requested category-specific breakdown reveals lower success on animation than timeline-editing tasks across all evaluated models. Coverage adds another distinction: GPT-6 Luna achieves 74.86% request coverage and 99.57% preservation coverage on animation tasks, but only 46.45% task success. Together, these results reveal differences across editing capabilities and partial fulfillment within unsuccessful tasks. We have incorporated category-level findings into Section 6.4, the partial-fulfillment analysis into Section 6.2, and detailed results and analysis into Appendix D.4.1 (Tables 19–20).

We have updated the main results for the 1,453-task benchmark (Section 6.2, Table 1), added binding criteria and dataset distributions (Section 4.2, Appendix A.5, Tables 8–9), and defined the macro metrics (Section 5.3, Appendix C.1). Task-level analysis and editing examples appear in Appendices D.4.2 (Table 21) and D.4.4.

AuthorsOpenReview ↗

Official Rebuttal [2/7]

W1-2. The primary environment is also a simplified editor: the authors added 19 of the 48 operations, and several are exposed through property-panel controls.

Thank you for your comment. We use OpenCut-TL for scalable evaluation and Premiere for evaluation through a professional editing interface. The 19 added operations broaden OpenCut’s editing capabilities, while some property-panel controls make precise edits more direct than the interaction paths agents used in Premiere.

We report the two environments separately to capture both scalable task evaluation and performance through a professional interface. We have clarified this distinction in Section 6.5. The matched Premiere results, interaction examples, and subset selection criteria are discussed in Official Rebuttal [4/7] and Appendix D.5.

W1-4. The human baseline consists of graduate students with little or no editing experience, and skipped assignments count as failures.

Thank you for raising this concern. We added an experienced-participant baseline on the same 100 tasks, extending the human comparison beyond novice users. Five participants with video-editing experience performed one assignment per task.

Participant / modelSR (%)
Human (novice)64.50
Human (editing experience)71.00
GPT-6 Sol92.00
GPT-6 Luna76.00
UI-TARS-1.5-7B6.00
OpenCUA-7B9.00

The experienced group achieved 71.00% success, with no skipped assignments. For comparison, novice success was 64.50% with skips counted as failures and 69.35% among completed executions. Skipped novice assignments receive zero request and preservation coverage. We have added the experienced-participant baseline and clarified skip handling in Section 6.6 (Table 3) and Appendix D.6 (Table 24).

W1-5. The abstract says preservation failures are a major source of error, but 89 of GPT-6 Sol’s 100 failures are unmet requests, and its preservation coverage is 99.82%.

Thank you for your feedback. We revised the abstract’s emphasis to reflect the observed failure composition. On the expanded 1,453-task benchmark, 138 of GPT-6 Sol’s 154 failed episodes (89.6%) are request-only failures, with no preservation violations. Unmet requests therefore account for most of its failures.

The revised abstract emphasizes uneven performance across editing categories and partial fulfillment revealed by coverage metrics. Section 6.4 (Figure 3) distinguishes request-only failures from preservation and other failure types.

AuthorsOpenReview ↗

Official Rebuttal [3/7]

W2-1. The automatic judge is assumed correct. The human study asks whether instructions look meaningful, self-contained, and fluent. It does not test whether the judge accepts professionally valid projects and rejects invalid ones, including an alternative effect stack, a different keyframe realization, or an extra but reasonable change. The reference program is built from the same specification as the evaluation rules, so agreement with it does not validate the judge. The same model, GPT-6 Astra, both rewrites the instructions and audits those rewrites.

Q2-1. On a sample of final projects stratified by success, unmet requests, and preservation or unauthorized-change failures, what fraction of judge decisions does an experienced editor agree with? Please include editorially reasonable alternative projects.

Thank you for requesting this validation. We conducted an additional human evaluation of 200 final projects with five participants with video-editing experience, followed by a review of disagreements with the evaluator.

Human validation. Each project received two ratings. We reused the 200 tasks from the task-quality annotation study, whose sampling procedure appears in Appendix A.6, but asked participants to assess the resulting projects rather than instruction quality. We selected one recorded model episode per task, comprising 100 evaluator successes, 60 request-only failures, 20 preservation or unauthorized-change failures without unmet requests, and 20 failures involving both.

Agreement and disagreement analysis. Blinded human judgments agreed with the evaluator in 80.1% of determinate responses (318/397). We then examined all 79 disagreements against the instructions and before-and-after project states. Our author-led review identified 62 responses in which human raters overlooked explicit instruction violations that the evaluator detected. Examples included accepting an unchanged clip despite a replacement request, accepting bold text despite a request for normal weight, and overlooking unrequested changes to title styling or clip opacity.

Applying these author-led classifications to the ratings yields 95.7% agreement (380/397). The remaining 17 responses included differences over evaluation criteria, such as whether a one-frame timing deviation within the predefined tolerance should count as a failure. The initial 80.1% measures blinded human–evaluator agreement, while 95.7% summarizes the subsequent author-led analysis. These cases illustrate how explicit project-state checks can identify violations missed during human review. We have added the study design, agreement results, and disagreement analysis to Appendix C.2.

Reference validation and alternative solutions. The reference program provides one executable solution for validating task construction. The evaluator checks task requirements rather than matching the reference project. For example, B-roll can occupy two overlay tracks instead of the reference’s single track when track placement is unspecified, provided the required source ranges, timing, narration, and audio settings are satisfied. This allows alternative valid project structures. All 1,453 tasks passed reference-based validation, which we describe as a specification–reference–evaluator consistency check in Section 4.2 and Appendix A.5.

Instruction verification. GPT-6 Astra’s author and critic roles revise and check instruction wording against a fixed task specification. As reported in Section 4.3, human annotators evaluated the resulting instructions for meaningfulness, self-containedness, and language quality. We also conducted an additional assessment of practical editing purposes with participants with video-editing experience.

Reviewer_XEgaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TimelineBench, a synthetic benchmark for evaluating video-editing agents. It constructs 781 tasks from 121 templates covering 48 atomic editing operations. The experiments evaluate 7 agents with a GUI harness, supplemented by human validation of task quality and cross-editor evaluation in professional software like Adobe Premiere Pro. The results show substantial differences in agent performance and highlight challenges in timeline manipulation and cross-editor generalization.

Critical strengths

Unlike benchmarks that primarily evaluate rendered videos, TimelineBench evaluates whether agents correctly modify project structures while preserving existing content and avoiding unintended changes.

The evaluation on professional software provides valuable insights into the agents' performance in realistic settings.

The proposed task construction pipeline appears scalable.

Critical weaknesses

The performance gap between the real and simulation environments raises concerns about generalization to real-world professional editing software. The results also indicate this point.

The paper does not clearly establish how agents would perform with more mature agent harnesses like Codex.

Questions for authors

747 out of 781 tasks contain only one to four atomic operations. Why are most tasks relatively short? This may limit the benchmark's ability to evaluate agents on complex, long-horizon video-editing workflows. In other words, if tasks are similar in the formalization, maybe 1/10 tasks can represent the whole benchmark.

5 comments
AuthorsOpenReview ↗

Official Rebuttal [1/3]

We thank the reviewer for recognizing the value of project-state evaluation, evaluation in professional editing software, and the scalability of our task construction pipeline.

W1. The performance gap between the real and simulation environments raises concerns about generalization to real-world professional editing software. The results also indicate this point.

Thank you for raising this concern. We included both editors to assess complementary aspects of video-editing agents. OpenCut-TL enables evaluation at scale, while Premiere tests agents through a professional editing interface. The matched comparison applies the same editing requests and project-state requirements in both editors, showing how agents perform in each setting.

On the same 56 tasks, Sol achieves 83.93% success in OpenCut-TL and 62.50% in Premiere, while Luna achieves 58.93% and 48.21%, respectively. These results show that execution performance varies across editors even when the requested editing outcomes are shared.

To understand what differed in execution across the two interfaces, we reviewed all 21 Premiere episodes scored as failures for Sol. We observed mismatches involving source selection, source intervals and durations, incomplete insertions, and content changes during clip movement. The following two tasks, completed successfully in OpenCut-TL but unsuccessfully in Premiere, illustrate how the same editing requests can lead to different execution outcomes.

  • Slide and slip. In OpenCut-TL, Sol entered offsets of 24 and −48 frames in the editing panel and succeeded. In Premiere, it switched tools and dragged on the timeline. The timeline placement was correct, but the source passage shifted in the opposite direction, and the agent declared completion.
  • Clip movement. In OpenCut-TL, Sol used the offset-movement menu to move both clips 24 frames later. In Premiere, repeated dragging and undoing reached the 80-action limit. Although the start positions were correct, the final clip lost 24 source frames and became shorter.

These cases illustrate difficulties in executing editor-specific edits and verifying that source content and preservation requirements remain satisfied. Reporting both settings therefore complements the large-scale results with direct evidence of agent performance in professional editing software.

We have clarified the roles of the two editors and the interpretation of the cross-editor results in the revised manuscript (Section 6.5), and added the matched execution cases in Appendix D.5.

AuthorsOpenReview ↗

Official Rebuttal [2/3]

W2. The paper does not clearly establish how agents would perform with more mature agent harnesses like Codex.

Thank you for pointing out this issue. We extended the evaluation to four models using agent implementations from the established OSWorld framework [1], allowing us to examine their performance beyond the native configurations.

We adapted the official implementations to TimelineBench while retaining model-specific prompting and action parsing. All four models received the same GUI tools, full visible text history, the five most recent screenshots, up to 4,096 output tokens per response, and an 80-model-call budget.

The comparison below uses the same 200 tasks. Native results use one trial per task with an 80-GUI-action budget. Shared-harness results report the mean and sample standard deviation across three trials with an 80-model-call budget.

ModelNative-harness SR (%)Shared-harness SR (%, mean ± std)
GPT-6 Sol83.0069.17 ± 1.04
GPT-6 Luna70.5056.17 ± 1.89
UI-TARS-1.5-7B8.504.67 ± 0.58
OpenCUA-7B3.503.83 ± 0.76

Native configurations achieve higher success rates for three of the four models, showing that the additional framework does not uniformly improve performance. Both GPT models nevertheless outperform both open-weight agents in every shared-harness trial. Thus, absolute success rates vary across the evaluated configurations, while the broad GPT–open-weight performance gap persists. These comparisons evaluate the complete configurations, including their different budget units.

We have added the additional evaluation and its settings to the revised manuscript (Section 6.3, Appendix D.3, Table 17).

[1] Tianbao Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS, 2024. https://arxiv.org/abs/2404.07972

AuthorsOpenReview ↗

Official Rebuttal [3/3]

Q1. 747 out of 781 tasks contain only one to four atomic operations. Why are most tasks relatively short? This may limit the benchmark's ability to evaluate agents on complex, long-horizon video-editing workflows. In other words, if tasks are similar in the formalization, maybe 1/10 tasks can represent the whole benchmark.

Thank you for the feedback on task complexity and diversity. We expanded the benchmark to include 130 tasks requiring at least five atomic operations, up from 34, and broadened the editing conditions evaluated within each template.

We construct each task around a concrete editing goal drawn from a tutorial, such as adjusting a shot or adding supporting footage. Many of these goals involve only a few operations. To cover more involved workflows, we generated additional instances of existing longer templates using different initial states, media, targets, and parameters.

Table 1. Task distribution by atomic operation count before and after expansion.

Number of atomic operationsOriginal 781Additional 672Expanded 1,453
1303381684
216683249
320727234
47185156
5 or more3496130

The share of tasks containing five or more operations increased from 4.35% to 8.95%. These longer compositions expose execution difficulties even for the strongest model. GPT-6 Sol succeeds on 56.25% of eight-operation tasks and 50.00% of nine-operation tasks, compared with 95.91% on single-operation tasks. Success is not monotonic with operation count, as the groups differ in operation composition and sample size (Appendix D.4.2, Table 21).

Multiple instances also let us evaluate the same editing workflow under different execution conditions. Different initial timelines, source passages, targets, and parameter values change what the agent must identify and edit, even when the operation sequence is shared. Each additional instance must therefore differ from every previously retained instance of the same template along at least two binding dimensions. This broadens coverage beyond a single realization of each workflow.

We have updated the main results to reflect evaluation on the expanded 1,453-task benchmark (Section 6.2, Table 1). We have also clarified the construction of longer tasks in the revised manuscript (Section 4.2) and added the final composition-length distribution and task binding criteria (Appendix A.5, Table 9).

Reviewer_kNNWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TimelineBench, a benchmark designed to evaluate AI agents for video editing. It addresses the limitations of traditional evaluation methods, which struggle to verify underlying project states, the preservation of original content, and the occurrence of unauthorized modifications. The authors extracted 165 workflows from 45 official tutorials for three mainstream video editing software programs, defining 48 atomic operations and 121 task templates to generate a total of 781 tasks. By comparing the structured project states before and after editing, the evaluator assesses task completion, state preservation, and unauthorized changes—without requiring agents to follow a predetermined sequence of actions. The authors also developed OpenCut-TL and conducted cross-platform validation using Adobe Premiere Pro. Experiments evaluating seven different agents revealed success rates ranging from 3.46% to 87.20%. The top-performing model, GPT-6 Sol, achieved an 87.20% success rate on OpenCut-TL, which dropped to 62.50% when tested on Premiere Pro; meanwhile, two open-weight agents both yielded success rates below 5%. These results demonstrate that the benchmark can effectively highlight performance disparities among agents and reveal their limitations in complex editing tasks. However, further validation is required regarding the accuracy of the evaluator itself and the extent to which the benchmark accurately represents real-world, complex editing projects.

Critical strengths

Process-based evaluation has been incorporated into the assessment of agent-driven video operations, with simultaneous checks on final execution outcomes, state preservation, and unauthorized modifications. The dataset construction pipeline offers good reproducibility and provides evaluations that cover a wide range of editing capabilities.

Critical weaknesses

The paper mentions that the 781 tasks in the final dataset are derived from 121 templates. While this template-based parameterization effectively increases the number of instances, it does not sufficiently demonstrate how well these tasks cover real-world scenarios or whether they align with actual editing needs. While the total number of instances is substantial, there is a significant disparity in the number of instances across different task categories. Consequently, the overall success rate is primarily driven by categories with a larger number of tasks, raising concerns regarding the stability of the test results for individual categories. Would it be possible to further expand the datasets for categories with fewer instances—for example, the "markers" category?

Questions for authors

The paper compares the performance of the closed-source GPT series models with two open-source models. However, the former utilized a "computer tool" provided by OpenAI, while the specific evaluation harness used for the latter is not specified. Given that the tools available to an agent significantly influence the final outcomes, are the results for these two categories of models truly comparable?

5 comments
AuthorsOpenReview ↗

Official Rebuttal [1/3]

We thank the reviewer for recognizing our reproducible task construction pipeline and project-state evaluation.

W1. The paper mentions that the 781 tasks in the final dataset are derived from 121 templates. While this template-based parameterization effectively increases the number of instances, it does not sufficiently demonstrate how well these tasks cover real-world scenarios or whether they align with actual editing needs.

Thank you for raising this concern. We conducted an additional annotation study to assess whether the benchmark tasks reflect practical editing needs. Five participants with video-editing experience judged whether they would perform or request each edit, with positive responses in 87.8% of ratings (351/400). The assessment reused the same 200 tasks from the original task-quality annotation study, with one or two instances per template and two ratings per task.

These tasks were designed around concrete editing goals drawn from official tutorials. For example, B-roll coverage illustrates narration while preserving its audio and timing, while music ducking lowers background music during dialogue without changing its level elsewhere. These requests connect editing operations to a purpose within an existing project and specify both what should change and what should remain unchanged.

The tutorial-grounded design and additional participant judgments together support the tasks’ relevance to practical editing needs. We have added the experienced-participant assessment to the revised manuscript (Section 4.3), with the shared task sampling procedure in Appendix A.6.

AuthorsOpenReview ↗

Official Rebuttal [2/3]

W2. While the total number of instances is substantial, there is a significant disparity in the number of instances across different task categories. Consequently, the overall success rate is primarily driven by categories with a larger number of tasks, raising concerns regarding the stability of the test results for individual categories. Would it be possible to further expand the datasets for categories with fewer instances—for example, the "markers" category?

Thank you for suggesting broader category coverage. In response, we expanded the dataset to provide at least 100 distinct task bindings per editing category and added macro success rates to assess the influence of unequal task counts.

First, we added 672 tasks and expanded TimelineBench from 781 to 1,453 tasks, prioritizing underrepresented categories. Each addition instantiates an existing tutorial-grounded template but must differ from retained instances of that template along at least two binding dimensions, so that the expansion covers varied editing conditions rather than near-duplicate instances.

Table 1 summarizes the resulting category distribution.

Table 1. Number of tasks per category before and after expansion.

CategoryOriginal 781Additional 672Expanded 1,453
Timeline editing438338776
Text274110384
Animation5699155
Audio2386109
Transitions2383106
Visual properties8226108
Playback1493107
Markers799106

Each category includes tasks containing at least one corresponding request condition. A task may belong to multiple categories.

The markers category highlighted in the review increased from 7 to 106 tasks. Other previously small categories, including playback, audio, and transitions, now also contain more than 100 tasks each. The expansion broadens evaluation beyond the few instances previously available in these categories.

Second, to account for the differences in category sizes, we compared pooled success rates with category- and template-macro success rates on the same 1,453-task evaluation results. The macro metrics assign equal weight to each category or template, respectively, with template macro assigning equal weight to each of the 121 task templates.

Table 2. Success rates under different aggregation methods on the expanded 1,453-task dataset.

ModelPooled (%)Category macro (%)Template macro (%)
GPT-6 Sol89.4090.5488.48
GPT-5.6 Sol84.6584.3985.80
GPT-6 Luna75.2979.9274.42
GPT-5.6 Luna71.7176.6071.44
GPT-5.4 mini31.0427.2039.92
OpenCUA-7B3.443.733.85
UI-TARS-1.5-7B3.172.374.58

These comparisons show that the ordering of the top five models is preserved across all three aggregation methods, with both Sol models’ macro success rates remaining within 1.15 percentage points of their pooled rates. The effect of weighting nevertheless differs across models: GPT-5.4 mini rises from 31.04% pooled success to 39.92% template-macro success, a difference of 8.88 percentage points, and the two open-weight models reverse order under template macro. We report the three aggregations together to expose distribution-sensitive comparisons (Appendix D.2, Table 16).

We have updated the main results to reflect evaluation on the expanded 1,453-task benchmark (Section 6.2, Table 1). We also added the final dataset’s binding criteria, validation procedure, and distributions (Section 4.2, Appendix A.5, Tables 8–9), together with macro metric definitions and the aggregation comparison (Sections 5.3 and 6.2, Appendices C.1 and D.2, Table 16).

AuthorsOpenReview ↗

Official Rebuttal [3/3]

Q1. The paper compares the performance of the closed-source GPT series models with two open-source models. However, the former utilized a "computer tool" provided by OpenAI, while the specific evaluation harness used for the latter is not specified. Given that the tools available to an agent significantly influence the final outcomes, are the results for these two categories of models truly comparable?

Thank you for pointing out this issue. We addressed the configuration differences in two ways. First, we re-evaluated the open-weight agents under their native protocols with expanded history and output limits. Both agents received full text action history, the five most recent screenshots, and up to 4,096 output tokens per response. Second, we evaluated four models using a shared OSWorld-based harness [1] to compare performance with the same GUI tools, visible-history policy, output limit, and model-call budget.

We preserved each model’s prompting and action format, using model-specific parsers to map its outputs to the shared GUI tools. Both columns below evaluate the same 200 tasks. Native results use one trial per task with an 80-GUI-action budget, while shared-harness results report the mean and sample standard deviation across three trials with an 80-model-call budget.

ModelNative-harness SR (%, 200 tasks)Shared-harness SR (%, 200 tasks, mean ± std)
GPT-6 Sol83.0069.17 ± 1.04
GPT-6 Luna70.5056.17 ± 1.89
UI-TARS-1.5-7B8.504.67 ± 0.58
OpenCUA-7B3.503.83 ± 0.76

Performance changes across agent configurations. Native configurations achieve higher success rates for three of the four models. However, both GPT models outperform both open-weight agents in every shared-harness trial. The shared comparison therefore supports the broad GPT–open-weight performance gap under matched interaction settings, while the differences between native and shared results show that absolute success rates depend on the evaluated configuration.

We have updated the main results in the revised manuscript (Section 6.2, Table 1), documented the expanded native settings (Section 6.1, Appendix D.3, Table 18), and added the shared-harness evaluation (Section 6.3, Appendix D.3, Table 17).

[1] Tianbao Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS, 2024. https://arxiv.org/abs/2404.07972

Comments on the paper

3 comments
AuthorsOpenReview ↗

Review Summary [3/3]

Strengths recognized by the reviewers

Lastly, we summarize the key strengths of our work using quotes from the reviewers.

Systematic evaluation of editable project states

  • “simultaneous checks on final execution outcomes, state preservation, and unauthorized modifications.” (kNNW)

  • “TimelineBench evaluates whether agents correctly modify project structures while preserving existing content and avoiding unintended changes.” (XEga)

  • “Scoring uses the saved project: the requested edit must be present, protected content must be unchanged, and no extra change is allowed.” (Kaz1)

Reproducible and scalable task construction

  • “The dataset construction pipeline offers good reproducibility” (kNNW)

  • “The proposed task construction pipeline appears scalable.” (XEga)

  • “The 781 tasks come from official Premiere Pro, Final Cut Pro, and DaVinci Resolve tutorials, organized as 48 edit operations and 121 templates.” (Kaz1)

Broad evaluation of editing capabilities and professional-software performance

  • “provides evaluations that cover a wide range of editing capabilities.” (kNNW)

  • “The evaluation on professional software provides valuable insights into the agents' performance in realistic settings.” (XEga)

  • “The full run is in OpenCut-TL, a modified version of OpenCut. A 56-task subset is repeated in Premiere Pro. Annotators rate 200 instructions, and ten novices attempt 100 tasks.” (Kaz1)

Empirical comparisons across agents and human participants

  • “Seven agents are tested once each. Success on OpenCut-TL ranges from 3.5% to 87.2%. On the novice subset, GPT-6 Sol scores 92%, compared with 64.5% for the novices. On the Premiere subset, Sol drops from 83.9% to 62.5%.” (Kaz1)

These quotations refer to the original submission. The revised manuscript expands the benchmark and strengthens these evaluations with the additional studies and analyses summarized above.

Thank you.

AuthorsOpenReview ↗

Review Summary [2/3]

Dataset coverage and interpretation of performance

  • Underrepresented categories (kNNW, Kaz1). Reviewers questioned sparse coverage of categories such as markers and animation. We added 672 tasks, expanding the benchmark from 781 to 1,453 tasks and bringing every editing category to at least 100 tasks. Markers increased from 7 to 106 tasks and animation from 56 to 155, directly strengthening the previously underrepresented categories (Section 4.2, Appendix A.5).

  • Annotation sampling and filtering (Kaz1). The reviewer asked how the 200 annotated tasks were selected and whether unfavorable judgments led to task removal. We documented coverage of all 121 templates, with one or two instances per template, and clarified that annotation judgments were not used to filter the dataset (Section 4.3, Appendix A.6).

  • Longer workflows (XEga, Kaz1). Reviewers questioned the limited coverage of longer workflows. We increased tasks requiring at least five operations from 34 to 130. GPT-6 Sol succeeds on only 56.3% of eight-operation tasks, showing that the expanded benchmark includes workflows that remain challenging even for the strongest model (Section 4.2, Appendix D.4.2).

  • Are the findings driven by task imbalance? (kNNW, Kaz1) We added success and both coverage metrics with equal category and template weighting, together with separate timeline-editing and animation results. The top five models retain their ordering, although weighting affects individual scores, including an 8.88-point increase for GPT-5.4 mini under template macro. These analyses establish which comparisons remain stable and expose capability differences hidden by the pooled average (Sections 6.2 and 6.4, Appendices C.1, D.2, and D.4).

  • Failure interpretation and claims (Kaz1). The reviewer challenged the original emphasis on preservation failures. Our expanded analysis shows that 138 of GPT-6 Sol’s 154 failures involve unmet requests without preservation violations. We removed the unsupported emphasis and revised the Abstract and Conclusion to reflect uneven editing capabilities and remaining improvement opportunities (Abstract, Section 6.4, Conclusion).

Agent configurations and repeated evaluation

  • History and output limits (kNNW, Kaz1). Reviewers questioned whether restricted settings disadvantaged open-weight agents. We re-evaluated both on all 1,453 tasks with expanded history and output limits. UI-TARS improved from 2.62% to 3.17%, and OpenCUA from 2.48% to 3.44%. The main results now use these expanded configurations, and both agents remain well below the GPT models (Section 6.1, Appendix D.3).

  • Does the gap persist under a shared harness and repeated trials? (kNNW, XEga, Kaz1) We evaluated four models using an OSWorld-based harness with shared GUI tools, visible-history policies, output limits, and model-call budgets across three trials on the same 200 tasks. Both GPT models outperform both open-weight agents in every trial, showing that the broad performance gap persists beyond the original native configurations and single runs (Section 6.3, Appendix D.3).

Project-state evaluation and professional-software experiments

  • What do project-state checks reveal beyond visual judging? (Kaz1) We compared render-only visual judgments with project-state evaluation on 200 episodes and analyzed their 18.5% binary disagreement rate. The analysis identified state information unavailable in rendered frames, such as an unrequested clip split, as well as errors in interpreting visible evidence, such as title timing. These cases demonstrate why rendered observations alone cannot reliably establish all editing requirements (Section 5.4, Appendix C.3).

  • Premiere performance and subset selection (XEga, Kaz1). Reviewers questioned the cross-editor performance gap and selection of the 56-task subset. We documented selection of one instance from each Premiere-XML-evaluable template and reviewed all 21 GPT-6 Sol failures in Premiere. Matched cases identify difficulties executing editor-specific interactions and verifying source content and preservation. The revision explains these execution differences and reports performance separately in each environment, rather than implying equivalent performance across editors (Section 6.5, Appendix D.5).

  • Exporter/parser consistency (Kaz1). The reviewer questioned agreement between the two project-state pipelines. We audited 46 execution pairs that succeeded in both editors. Compared properties matched directly in 40 pairs, and review of the remaining six identified task-permitted editing differences or representation differences. We found no confirmed exporter/parser errors in the examined properties, supporting consistency within the audited scope (Appendix D.5).

AuthorsOpenReview ↗

Review Summary [1/3]

Dear Area Chair,

We have responded to every question and concern raised by the reviewers through additional experiments, human studies, analyses, and clarifications, and incorporated these changes into the submitted revision. We conducted three additional studies with participants who have video-editing experience, expanded the benchmark, and added shared-harness evaluations with repeated trials. Reviewer Kaz1 subsequently indicated that they “now lean toward weak accept.” Reviewer kNNW also posted a follow-up. As its wording resembled an author’s response to a reviewer, we asked for clarification on whether it was intended as a reply to our rebuttal or might have been posted by mistake, and invited any remaining concerns. Below, we summarize how each concern has been addressed.


Additional human studies

  • Evaluator validity (Kaz1). The reviewer requested validation on completed projects rather than instruction quality alone. Five participants with editing experience evaluated 200 final projects, with initial blinded agreement of 80.1%. We closely examined all 79 disagreements against the instructions and before-and-after project states. In 62 cases, the evaluator correctly detected explicit instruction violations that human raters had missed, bringing agreement to 95.7% under our detailed review. These findings show that the evaluator catches concrete editing violations that can escape human inspection (Appendix C.2). We also clarified that reference execution checks specification–reference–evaluator consistency, while the evaluator accepts alternative project structures that satisfy the task requirements rather than requiring a match to the reference project.

  • Practical task relevance (kNNW). The reviewer questioned whether the tasks reflect actual editing needs. Five participants with editing experience judged whether they would perform or request each of 200 edits. Positive responses accounted for 87.8% of ratings (351/400), providing direct evidence that the tasks address practical editing needs (Section 4.3).

  • Instruction verification (Kaz1). The reviewer questioned using the same model to rewrite and check instructions. We clarified that both roles operate against a fixed task specification, while separate human annotation assesses the resulting instructions for meaningfulness, self-containedness, and language quality (Section 4.3, Appendix B).

  • Experienced human baseline and skips (Kaz1). The reviewer questioned the novice-only baseline and the treatment of skipped assignments as failures. We added five participants with editing experience on the same 100 tasks. They completed every assignment without skips and achieved 71.0% success, extending the comparison beyond novices with a baseline unaffected by skipped assignments. We also explicitly documented skip scoring for the novice results (Section 6.6, Appendix D.6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.