acceptodds
Under review as a conference paper at ICLR 2027

Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic Tasks in Real-World Professional Fields

Abstract

Recent advances in AI agents have enabled increasingly complex interactions with real-world environments, yet existing GUI benchmarks largely focus on general-purpose software and short-horizon tasks. It therefore remains unclear whether modern agents can autonomously operate domain-specific professional software to complete long-horizon, economically valuable workflows end to end. To address this gap, we introduce Workflow-GYM, a benchmark for evaluating GUI agents on realistic professional workflows across diverse domains and specialized software environments. Extensive experiments show that even the strongest models achieve only slightly above 30% success rates, revealing substantial limitations in current GUI agents. Further analysis identifies several recurring failure modes, including workflow stage omission, error propagation, objective drift, and insufficient understanding of professional software environments. These results highlight long-horizon workflow consistency and reliable interaction with specialized software as key challenges for the next generation of GUI agents.

Primary area:
datasets and benchmarks
Keywords:
long-horizon task, computer-use agent, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S4ryOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that maintaining coherent workflow execution and reliably operating specialized software remain major challenges for current GUI agents and introduces Workflow-GYM, a benchmark designed to evaluate computer-use agents on long-horizon workflows involving specialized professional software. The benchmark contains 338 tasks across six broad domains, 23 subdomains, and 56 software environments, which are sourced from domain experts and constructed through a multi-stage process involving environment setup, expert demonstrations, instruction validation, etc.

The authors evaluate six representative models, each using a compatible computer-use framework, with three independent trials per task. Additional analyses examine the relationship between workflow length and task difficulty, categorize common failure modes, and assess the effects of textual and visual procedural guidance.

Critical strengths
  1. I think a major strength is the scale and diversity of the benchmark. Constructing 338 executable tasks across 56 professional software environments, including both Windows and Ubuntu applications, requires substantial engineering and annotation effort. It could become a useful resource if the environments and evaluation procedures are made very accessible.

  2. It also has very useful empirical analysis beyond aggregate success rates, which includes repeated trials, domain-level results, workflow-length analysis, and procedural-guidance ablations. The analysis further illustrates concrete problems that arise during extended GUI execution.

Critical weaknesses
  1. One of my concern is the reliability of the benchmark's success labels is not sufficiently established. The benchmark relies heavily on model-based evaluation. According to Appendix D, 168 of the 338 tasks are evaluated through VLM-based inspection of the final GUI state, while another 102 use LLM-based artifact evaluation. Thus, approximately 80% of the tasks depend on model-based judgments.
  2. The paper positions Workflow-GYM as addressing a largely unexplored intersection of professional expertise, long-horizon execution, real GUI interaction, and economically meaningful outcomes. However, related benchmarks already investigate several important parts of this intersection such as OSWorld 2.0 and WeaveBench... I think the distinction from existing long-horizon professional workflow benchmarks needs stronger justification.
Questions for authors
  1. For the tasks using VLM/LLM-based verification, could the authors report human agreement on complete binary task outcomes rather than only individual criteria?
  2. How does Workflow-GYM differ substantively from recent long-horizon computer-use benchmarks?
  3. What are the concrete release and reproducibility plans for the benchmark?
2 comments

We thank reviewer S4ry for providing insightful feedback. Below we provide point-by-point responses to the concerns raised:

Weakness 1 and Question 1

We thank reviewer for raising this important concern. Workflow-GYM emphasizes complex, end-to-end deliverables in real-world professional scenarios, where the required outcomes often involve visual semantics, spatial relationships, and domain-specific constraints that are difficult to comprehensively verify through deterministic rules. For example, one task requires agents to create a multi-layer 2D town map in Tiled, placing three types of characters in specific spatial relationships to a blue building (three to its left, three below it, and four inside it). Reliably verifying such requirements requires identifying visual entities and interpreting their spatial relationships, which are difficult to generalize through rule-based verification alone. Therefore, we adopt rule-based evaluation wherever feasible and employ model-based evaluators when necessary to verify expert-defined criteria.

Regarding evaluation reliability, as reported in Appendix D, we conducted a human–model agreement study on over 100 sampled evaluation cases, covering both VLM-based GUI-state evaluation and LLM-based artifact evaluation, achieving over 90% agreement. We clarify that this agreement is measured on binary task-level outcomes rather than individual criteria. Although exhaustive human verification of all task executions would be prohibitively expensive, our sampling-based validation provides empirical evidence supporting the reliability of model-based evaluation. We will clarify the agreement measurement and validation protocol in the revised version.

Weakness 2 and Question 2

We thank reviewer for raising this important point, and would like to clarify two aspects:

1. Contemporaneous work. Compared with Workflow-GYM, OSWorld 2.0[1] WeaveBench[2], were independently developed and publicly released around the same time in June 2026. According to the ICLR 2027 Reviewer Guidelines, authors are not required to compare against contemporaneous or unpublished arXiv works. Nevertheless, we appreciate the reviewer's suggestion and will incorporate a discussion of these related benchmarks in the revised version.

2. Distinct research objectives. Despite their shared focus on long-horizon computer use, these benchmarks address fundamentally different research questions. OSWorld 2.0 focuses on challenging interaction and reasoning phenomena in real-world computer-use workflows, while WeaveBench emphasizes cross-interface orchestration involving GUI, CLI, and code execution. In contrast, Workflow-GYM specifically investigates whether GUI agents can accomplish economically valuable, long-horizon workflows in real-world professional domains. Its systematic coverage of 338 tasks across 56 specialized software environments enables the evaluation of domain-specific knowledge, professional software proficiency, and long-horizon GUI execution in an integrated manner. We therefore view these benchmarks as complementary rather than redundant.

As shown in table below, although all 3 benchmarks investigate long-horizon computer use, they differ substantially in task selection principles, interaction paradigms, and targeted agent capabilities. Workflow-GYM uniquely emphasizes the systematic evaluation of economically valuable professional workflows through specialized GUI software.

DimensionOSWorld 2.0WeaveBenchWorkflow-GYM (Ours)
# Tasks108114338
Task Selection FocusReal-world workflows with challenging interaction phenomenaReal-world workflows requiring cross-interface collaborationEconomically valuable workflows in specialized professional domains
Interaction ParadigmGeneral computer useHybrid GUI + CLI/CodeGUI-only interaction with 56 professional software applications
Core Evaluated CapabilitiesCross-source reasoning, implicit-state inference, and complex interactionsCross-interface orchestration and state coordinationDomain-specific expertise, professional software proficiency, and long-horizon workflow execution

References

[1]Yuan M, Zhou Z, Xiong X, et al. OSWorld2. 0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks[J]. arXiv preprint arXiv:2606.29537, 2026.

[2]Li W, Zhou B, Yu Y, et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces[J]. arXiv preprint arXiv:2606.09426, 2026.

Question 3

We thank reviewer for raising this concern and agree the imortance of reproducibility of the results. To mitigate potential data contamination risks and preserve the reliability of benchmark evaluations, we have not yet publicly released the benchmark dataset. In the final version, we will publicly release a subset of Workflow-GYM, together with the corresponding execution environments and evaluation protocols, and report the performance of different models on this subset to facilitate independent evaluation and improve reproducibility.

Reviewer_hKefOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the capability of GUI agents to autonomously complete long-horizon workflow tasks inside professional software, and proposes the Workflow-GYM benchmark. It covers 338 tasks across 6 domains, 23 subdomains and 56 software environments. The benchmark adopts expert-authored workflows, virtual-machine sandboxes, three-stage task validation, and an evaluation paradigm based on final deliverables or GUI states.

Evaluations over six models and their compatible frameworks yield a peak average success rate of 30.67% and a top Pass@3 of 41.42%. All systems achieve substantially lower success rates on longer workflow groups. Failure breakdowns further reveal that weaker agents mostly fail to get through the workflow at all, while stronger ones still frequently produce end outputs that do not meet specifications.

Critical strengths
  1. By integrating professional software scenarios, long-step workflows, and final deliverable requirements, this paper enables the evaluation of cross-stage dependencies and result correctness.

  2. The benchmark boasts broad task coverage, including 338 tasks spanning 6 domains, 23 subdomains and 56 software environments. All tasks are executable on Windows and Ubuntu virtual machines, which means the constructed resources are not limited to a single application or domain. This provides a solid foundation for analyzing the cross-domain performance of intelligent agents.

  3. Each task is equipped with expert-authored reference workflows and undergoes rigorous validation, including real-world execution by experts, instruction review, and preliminary agent trial runs. Compared with datasets that only provide textual task descriptions, this validation mechanism significantly improves the credibility and feasibility of task execution in the designated environments.

Critical weaknesses
  1. Since no adapted evaluation harness is proposed, it is difficult to achieve fully fair model comparison.

  2. The failure classification and behavioral analysis are not sufficiently in-depth, and fail to provide strategic guidance for the improvement of Computer Use Agent (CUA) capabilities.

  3. This benchmark lacks a characteristic comparison with existing homogeneous CUA benchmarks.

Questions for authors
  1. How are the rubrics for the LLM Judger constructed? Are there unified rubric design criteria across different tasks to mitigate biases introduced by human experts?

  2. What are the core differences and unique values compared with existing Computer Use agent benchmarks? A table summarizing the characteristics of related benchmarks would strengthen the comparison.

  3. Why does the evaluation focus solely on final result correctness? Especially given the available failure diagnosis and expert reference strategy, why are process-level metrics not introduced to enhance benchmark evaluation capability, such as execution efficiency and redundant action ratios?

3 comments

We thank reviewer hKef for providing insightful feedback and constructive suggestions. Below we provide point-by-point responses to the concerns raised:

Weakness 1

We thank reviewer for raising this important concern regarding the fairness of model comparison. We would like to clarify that our evaluation already incorporates model-specific interface adaptations while maintaining a unified agentic workflow. Specifically, all models follow the same GUI interaction paradigm: observing screenshots, generating GUI actions, executing these actions in isolated environments, and iteratively receiving updated observations. The task environments, interaction budgets (400 turns), and evaluation criteria remain consistent across all models. Model-specific adaptations primarily concern tool schemas, coordinate representations, screenshot resolutions, and other interface configurations, rather than the underlying agent architecture.

Importantly, enforcing an identical harness configuration across all models does not necessarily ensure a fairer comparison. For example, Gemini models use a normalized coordinate system (0–1000) for GUI actions. Without appropriate coordinate conversion, even correctly intended actions could fail, resulting in performance degradation attributable to interface incompatibility rather than the model's actual GUI capabilities.

Therefore, our evaluation protocol aims to balance consistency in the core evaluation framework and compatibility with model-specific interaction conventions. These adaptations eliminate artificial interface-related disadvantages without introducing additional reasoning, planning, or execution capabilities. We believe this design enables a fairer and more meaningful comparison across models. We will further clarify these implementation details and their rationale in the revised version.

Weakness 2

We thank reviewer for this insightful comment and fully agree that an important purpose of benchmark-driven failure analysis is to provide actionable guidance for future CUA development. We would like to emphasize that our findings offer such guidance at two complementary levels:

  1. Model-level capability improvement. The failure modes identified in Workflow-GYM reveal specific capability bottlenecks that can inform future model training. For instance, error propagation highlights the importance of intermediate state verification and error recovery; workflow stage omission suggests the need for improved long-horizon planning and progress tracking; objective drift calls for stronger goal consistency and long-horizon instruction following; and software knowledge deficiency motivates training on professional software knowledge and domain-specific operational procedures. These findings can directly inform training data construction and the design of targeted training objectives.

  2. Agent-level architectural improvement. Beyond model-level deficiencies, our analysis identifies a more fundamental mismatch between the discrete observation-action paradigm adopted by current CUAs and the continuous visual observation required by certain real-world workflows. This finding suggests that improving model capabilities alone may not be sufficient, and motivates alternative CUA architectures incorporating continuous visual perception and asynchronous environment monitoring.

Therefore, we believe that our failure analysis goes beyond describing unsuccessful behaviors and provides actionable insights for both model training and CUA architectural design. We will further elaborate on these implications in the revised version.

Weakness 3 and Question 2

We thank the reviewer for this constructive suggestion and agree that a systematic comparison with closely related CUA benchmarks would help clarify the distinctive characteristics of Workflow-GYM. As summarized below, although these benchmarks share the goal of evaluating agents on complex real-world workflows, they differ in their task selection principles, interaction environments, and core evaluation focus. While OSWorld 2.0[1] emphasizes challenging interaction phenomena, WeaveBench[2] focuses on hybrid-interface orchestration, and SaaS-Bench[3] targets cross-application professional workflows within SaaS environments, Workflow-GYM specifically emphasizes economically valuable, long-horizon workflows that require both domain-specific expertise and proficiency in specialized professional desktop software. We believe these benchmarks provide complementary perspectives on CUA capabilities, and we will incorporate this comparison into the revised manuscript.

DimensionOSWorld 2.0WeaveBenchSaaS-BenchWorkflow-GYM (Ours)
# Tasks108114106338
Task Selection FocusReal-world workflows involving challenging interaction phenomenaReal-world workflows requiring cross-interface collaborationProfessional workflows involving coordination across SaaS applicationsEconomically valuable, long-horizon workflows in specialized professional domains
Interaction ParadigmGeneral computer useHybrid GUI + CLI/CodeBrowser-based interaction across 23 SaaS systemsGUI-only interaction across 56 professional software applications
Core Evaluated CapabilitiesCross-source reasoning, implicit-state inference, and complex interactionsCross-interface orchestration and state coordinationCross-application coordination, long-horizon planning, and system state managementDomain-specific expertise, professional software proficiency, and end-to-end workflow execution

References

[1]Yuan M, Zhou Z, Xiong X, et al. OSWorld2. 0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks[J]. arXiv preprint arXiv:2606.29537, 2026.

[2]Li W, Zhou B, Yu Y, et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces[J]. arXiv preprint arXiv:2606.09426, 2026.

[3]Shi K, Li Z, Ma T, et al. SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?[J]. arXiv preprint arXiv:2605.15777, 2026.

Question 1

We thank reviewer for raising this question. The rubrics are constructed by the domain experts who design the corresponding tasks, following shared design principles. Specifically, each rubric should be clearly stated and unambiguous, grounded in the task’s intended deliverable, and sufficiently comprehensive to determine whether the final artifact or GUI state meets the requirements for end-to-end completion. Since task success is evaluated in a binary manner, the rubric should support a clear pass/fail decision based on fulfillment of these requirements. Thus, while rubric content necessarily varies across professional domains and tasks, the underlying design criteria remain consistent. During pilot experiments, we also review the execution outcomes and judge assessments to identify and revise ambiguous, incomplete, or misaligned rubric criteria and judging instructions. This process helps reduce variation arising from individual experts’ interpretations and improve alignment between automated judgments and the intended delivery requirements. We will clarify these points in details in the revised version.

Question 3

We thank reviewer for this constructive suggestion and agree that process-level metrics are valuable for understanding agent behavior and complementing outcome-based evaluation.

  1. Outcome-based evaluation reflects the benchmark’s primary objective. Workflow-GYM evaluates whether agents can complete professional workflows end to end and produce deliverable artifacts or final states. Accordingly, we use binary task success to indicate whether the final outcome satisfies the delivery requirements. This directly captures the capability targeted by our benchmark, while process-level metrics provide complementary information about how efficiently and reliably agents achieve that outcome.

  2. Existing process analysis and planned extensions. The paper already includes quantitative analysis of repetitive action looping (§5.2), defined as producing the same action sequence for 10 consecutive interaction turns, together with its distribution across domains. We also analyze execution trajectory lengths under different guidance conditions (§5.4.2). Nevertheless, we agree that a more systematic evaluation of execution efficiency would strengthen the benchmark. In the revised version, we will expand this analysis by reporting average interaction turns, separately for successful and failed runs, and execution-efficiency comparisons. These procedures will serve as reference baselines rather than uniquely optimal paths, since valid workflows may follow different execution strategies.

Reviewer_3V9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Workflow-GYM, for evaluating GUI agents on realistic professional workflows across diverse domains and specialized software environments. It contains 338 expert-constructed professional GUI tasks spanning six domains and 56 software environments. Evaluations show substantial limitations in current agents, with further analysis examining workflow failures and the benefits of procedural guidance.

Critical strengths
  1. Valuable Data. The paper provides a valuable benchmark of 338 tasks constructed with 71 domain experts, covering six domains and 56 software environments for evaluating professional GUI workflows.
  2. Robust Evaluation. Each model is evaluated over three independent trials per task, reducing reliance on a single execution and providing a more robust assessment of performance.
Critical weaknesses
  1. The contribution beyond existing benchmarks is insufficiently established. Professional software and long-horizon GUI tasks have already been studied, but the paper does not clearly demonstrate what new evaluation capabilities or insights Workflow-GYM offers. Findings such as lower success rates on longer workflows and improvements from procedural guidance are largely expected.
  2. The experimental coverage and failure analysis are limited. The evaluation lacks dedicated GUI-agent baselines, while most behavioral failure categories are illustrated through selected cases without reporting their frequencies. This limits the evidence supporting the proposed explanations of agent failures.
Questions for authors
  1. It would be helpful to include results from dedicated open-weight GUI models and representative task examples at each difficulty level.
  2. What proportion of candidate tasks was filtered out at each construction stage? Were tasks independently reviewed by other experts, and what was the level of agreement among reviewers?
3 comments

We thank reviewer 3V9V for providing insightful feedback to our work. Below we provide point-by-point responses to the concerns raised:

Weakness 1

We thank reviewer for this comment. We respectfully disagree that Workflow-GYM offers limited contributions or that its main findings are largely expected. We would like to clarify both the distinctive evaluation capabilities of our benchmark and the empirical insights it provides.

  1. Distinctive evaluation scope and capabilities. While existing benchmarks have investigated professional software or long-horizon GUI tasks, Workflow-GYM systematically targets their intersection: end-to-end, domain-specialized professional software workflows. Unlike general-purpose computer-use benchmarks, Workflow-GYM explicitly requires every task to involve specialized professional software, domain-specific knowledge, and at least 30 atomic GUI operations. These are mandatory construction criteria rather than incidental characteristics of selected tasks. Our benchmark comprises 338 expert-constructed tasks across 56 professional software environments, with expert-verified execution procedures and domain-specific outcome requirements. This provides a dedicated evaluation setting for professional workflow execution, rather than simply extending the number of tasks in existing GUI benchmarks.

  2. Non-trivial empirical findings beyond expected trends. Our analyses go beyond showing that longer workflows are harder or that procedural guidance improves performance. In particular:

  • Capability-dependent failure patterns (Sec. 5.1). We observe a strong negative correlation between workflow incompletion rates and overall model performance (Pearson r = −0.97). More importantly, the failure composition changes with model capability: weaker agents predominantly fail to complete workflows, whereas stronger agents more frequently reach the final stage but produce incorrect outcomes. This reveals distinct bottlenecks at different capability levels, rather than merely a monotonic relationship between task length and difficulty.
  • A structural limitation of discrete GUI interaction (Sec. 5.3). We identify a mismatch between the discrete screenshot–action paradigm used by current GUI agents and professional operations requiring continuous visual feedback, such as geometric editing, object alignment, and precise dragging. This suggests that some failures cannot be addressed solely through better reasoning or planning, but may require more continuous, closed-loop perception–action mechanisms.
  • Complementary benefits of visual procedural guidance (Sec. 5.4). Our controlled ablations show that visual demonstrations provide additional benefits beyond textual step-by-step instructions. On 100 tasks, adding visual demonstrations improves success from 28% to 31% for Seed-2.0-lite and from 28% to 35% for Gemini-3.1-Pro. Moreover, substantial failures persist even with detailed guidance, indicating that execution-level limitations remain after reducing planning uncertainty.
  1. Relationship to contemporaneous benchmarks. We would like to respectfully note that the ICLR 2027 reviewer guildlines(https://iclr.cc/Conferences/2027/ReviewerGuidelines) explicitly state that comparisons with contemporaneous or non-peer-reviewed arXiv papers are not mandatory, and that the absence of such comparisons should not constitute grounds for rejection. Nevertheless, we acknowledge the relevance of these works and would like to clarify our distinctions. Workflow-GYM and OSWorld 2.0 are independently developed works with overlapping motivations but distinct evaluation focuses. While OSWorld 2.0 broadly investigates complex real-world computer use, Workflow-GYM specifically concentrates on end-to-end professional software workflows of high economic value and their domain-specific execution requirements. We do not claim that existing benchmarks contain no such tasks; rather, our contribution lies in their systematic coverage and analysis as a dedicated evaluation target. We will further clarify these distinctions in the revised manuscript.

Weakness 2

We thank reviewer for raising this concern and agree the significance of experimental coverage and failure analysis.

  1. Harness Comparability and Model-Specific Adaptations. We would like to clarify that, despite minor model-specific configurations, all evaluated models share essentially the same GUI interaction paradigm: observing screenshots, generating GUI actions, executing these actions in the environment, and iteratively receiving updated observations. The underlying agentic workflow, task environments, interaction budgets, and evaluation criteria remain consistent across models. The differences primarily concern tool schemas, coordinate representations, screenshot resolutions, and other model-interface parameters, rather than fundamentally different agent architectures. Therefore, we believe the experimental setups remain highly comparable.

More importantly, enforcing an identical harness configuration across all models does not necessarily lead to a fairer evaluation. Different models may have distinct interface conventions and input/output requirements. For example, Gemini models use a normalized coordinate system (0–1000) for GUI actions. If their outputs were directly interpreted as native pixel coordinates without appropriate conversion, their performance could be substantially degraded, even when the intended actions are correct. Such degradation would reflect interface incompatibility rather than the model's actual GUI reasoning and interaction capabilities.

Therefore, we adopt model-specific interface adaptations to ensure that each model can operate under its intended interaction conventions, while maintaining consistent task conditions and evaluation protocols. These adaptations serve to eliminate artificial interface-related disadvantages, rather than provide additional reasoning, planning, or execution capabilities. We believe this approach better reflects the models' actual GUI-agent capabilities and enables a more meaningful comparison. We will clarify these implementation details and the rationale for model-specific configurations in the revised version.

  1. Quantitative and Qualitative Failure Analysis. We would like to clarify that our paper already includes quantitative failure analysis in Section 5.1, where we categorize unsuccessful executions into Workflow Incompletion and Final-State but Incorrect, and report a strong correlation (Pearson (r=-0.97)) between workflow incompletion rate and overall performance. Beyond these outcome-level categories, Section 5.2 further identifies behavioral failure modes, including Error Propagation, Objective Drift, and Stage Omission. Unlike outcome-level failures, reliably identifying these behavioral patterns requires detailed examination of complete, long-horizon interaction trajectories and their task-specific contexts. Therefore, obtaining accurate frequency estimates across all trajectories is substantially more challenging than outcome-level categorization.

Nevertheless, we agree that quantitative evidence would strengthen these observations. As a preliminary step, we screened all failed trajectories from Kimi-K2.6 and found that approximately 30% exhibited potential signs of Objective Drift. (This is an initial screening result rather than a verified frequency, and the estimate may change substantially following more rigorous annotation.) We are conducting further trajectory-level analysis, with more extensive results to be incorporated into the revised version.

Question 1

We thank reviewer for raising this helpful suggestion.

  1. Dedicated open-weight GUI models. Our evaluation already includes Kimi-K2.6 as an open-weight baseline,a nd we also plan to select suitable dedicated open-weight GUI models for additional evaluation and, supplement the results in the revised version.

  2. Representative examples at each difficulty level. To make the benchmark tasks more concrete, we provide three examples below, with task descriptions summarized from the original specifications:

  • Easy — Wireframe cage construction in Blender. Starting from an empty scene, create a cube with dimensions of 4 × 4 × 3 m and a vertical location of 1.5 m. Configure a bevel modifier with an amount of 0.2 m, subdivide the mesh with 10 cuts, apply Poke Faces followed by Triangles to Quads, and add a wireframe modifier with a thickness of 0.03 m and an offset of −1. Enable Boundary and disable Even Thickness. Finally, display the scene statistics and verify the resulting object, vertex, edge, face, and triangle counts.
  • Medium — Closed-loop control simulation in OpenModelica. Construct a closed-loop control system consisting of a step reference, a PID controller, and a first-order plant. Connect the components to form the feedback loop, run the simulation using the default PID parameters, plot the system output, and export the output curve as a CSV file to the specified directory. The expert reference workflow contains 46 steps.
  • Hard — Gas mixing, compression, and cooling simulation in DWSIM. Using the Peng–Robinson property package and SI units, configure two pure feed streams—methane and ethane—each at 298.15 K, 101,325 Pa, and 1 kg/s. Mix the streams, compress the mixture to 200,000 Pa, and cool it to 293.15 K. Determine the compressor power consumption and cooler heat duty, then export the simulation results and convert them into an Excel file at the specified output path. The expert reference workflow contains 75 steps.

We will also include these examles in the revised version.

Question 2

We thank reviewer for raising this request.

  1. Task filtering across construction stages. We initially collected 1,308 candidate workflows, of which 338 were ultimately retained (25.84%). The table below summarizes the approximate exclusions across the construction process, using terminology corresponding to Section 3.1 and Appendix C. All exclusion percentages use the initial 1,308 candidates as the denominator, rather than the number entering each stage.
Construction stageValidation and filtering criteriaApproximate proportion excluded
Workflow Selection, Environment Verification, and Instruction-Level ValidationCandidates were screened for realism, domain specificity, complexity, and verifiability. Experts manually executed and recorded tasks in the prepared environments to verify solvability. Independent expert review and LLM-assisted instruction checks identified incomplete, ambiguous, or inconsistent specifications.35%
End-to-End ValidationSOTA GUI agents executed the tasks, and experts inspected their trajectories and outputs to identify latent ambiguities, missing constraints, and environment inconsistencies. Affected tasks were revised and revalidated, or excluded if the issues remained unresolved.30%
Iterative revalidation and stability checksRepeated testing further checked environment and evaluation stability. Tasks with unresolved reproducibility or stability issues were excluded. These checks formed part of iterative revalidation rather than a separate validation category.10%
Final inclusionTasks satisfying the selection criteria and passing all three validation checks were retained.338 retained (25.84%)
  1. Independent expert review. We employed a cross-review mechanism during quality control: each task was reviewed by at least one domain expert other than its author, following common criteria covering background sufficiency, instruction clarity, objective assessability of the outcome, and compatibility between the task and its environment. Reviewers provided feedback, and tasks with identified issues were returned for revision and re-reviewed. Only tasks that passed this cross-review process were included in the final benchmark.

We will append all these details into the revised version.

Comments on the paper

1 comment

Summary of Reviewer Concerns and Our Responses

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have provided detailed point-by-point responses to each review. Here, we summarize the key contributions of Workflow-GYM, the reviewers' primary concerns, and our corresponding clarifications.

1. Our Contributions

Workflow-GYM introduces a benchmark specifically designed to evaluate end-to-end, economically valuable, long-horizon GUI workflows in specialized professional domains, comprising 338 expert-constructed tasks across 6 domains and 56 professional software environments. Beyond providing a systematically validated benchmark with expert reference workflows, our experiments reveal important capability bottlenecks in current GUI agents, including capability-dependent failure patterns, limitations of discrete GUI interaction in tasks requiring continuous visual feedback, and the complementary benefits of visual procedural guidance.

2. Reviewers' Primary Concerns

  1. Novelty and distinctive contributions: Whether Workflow-GYM offers sufficiently distinct evaluation capabilities and novel empirical insights compared with existing long-horizon computer-use benchmarks.

  2. Benchmark reliability: Whether task construction and validation are sufficiently rigorous, whether model-based judgments reliably reflect human evaluations.

  3. Experimental fairness, and analytical depth: Whether model-specific harness configurations allow fair comparisons, and whether the failure analysis provides actionable insights for CUA development.

3. Our Responses to the Concerns

  1. Novelty and distinctive contributions. Workflow-GYM systematically targets the intersection of domain-specific professional knowledge, specialized GUI software, and long-horizon end-to-end execution. Unlike OSWorld 2.0, WeaveBench, and SaaS-Bench, which emphasize different aspects of complex computer use, Workflow-GYM specifically investigates agents' ability to autonomously complete economically valuable workflows across 56 specialized professional software environments. We have provided a detailed comparison table in our rebuttal.

Moreover, our empirical findings extend beyond expected observations about task difficulty. For example, we identify capability-dependent changes in failure composition (Pearson r = −0.97 between workflow incompletion rate and overall model performance), a structural mismatch between discrete GUI interaction and continuous visual-feedback requirements, and additional gains from visual demonstrations beyond textual guidance. These findings offer distinct insights into both model capability limitations and potential architectural improvements.

  1. Benchmark reliability and reproducibility. We clarify that Workflow-GYM underwent rigorous multi-stage validation. Of 1,308 initial candidate workflows, 338 were ultimately retained (25.84%). Each task was independently cross-reviewed by at least one additional domain expert, following shared quality criteria covering instruction clarity, background sufficiency, outcome assessability, and environment compatibility.

For evaluation reliability, we adopt rule-based verification wherever feasible and use model-based evaluation when complex artifacts or GUI states require semantic understanding. As clarified in our rebuttal, our human–model agreement study covers over 100 sampled evaluation cases and achieves over 90% agreement on binary task-level outcomes, supporting the reliability of model-based evaluation.

  1. Experimental fairness, coverage, and analytical depth. All evaluated models follow the same core screenshot–action interaction paradigm, with consistent task environments, interaction budgets (400 turns), and evaluation criteria. Model-specific adaptations are limited to interface compatibility, such as coordinate representations, tool schemas, and screenshot resolutions. These adaptations avoid artificial disadvantages caused by interface incompatibility rather than introducing additional reasoning or execution capabilities.

Moreover, our failure analysis provides actionable guidance at both the model level (e.g., improving long-horizon planning, progress tracking, error recovery, and domain-specific software knowledge) and the agent architecture level (e.g., incorporating continuous visual perception and asynchronous environment monitoring). We acknowledge that further quantitative analysis would strengthen these findings. Accordingly, we are conducting additional trajectory-level failure annotations and plan to extend the analysis of failure-mode frequencies and execution efficiency in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.