- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that maintaining coherent workflow execution and reliably operating specialized software remain major challenges for current GUI agents and introduces Workflow-GYM, a benchmark designed to evaluate computer-use agents on long-horizon workflows involving specialized professional software. The benchmark contains 338 tasks across six broad domains, 23 subdomains, and 56 software environments, which are sourced from domain experts and constructed through a multi-stage process involving environment setup, expert demonstrations, instruction validation, etc.
The authors evaluate six representative models, each using a compatible computer-use framework, with three independent trials per task. Additional analyses examine the relationship between workflow length and task difficulty, categorize common failure modes, and assess the effects of textual and visual procedural guidance.
Critical strengths
-
I think a major strength is the scale and diversity of the benchmark. Constructing 338 executable tasks across 56 professional software environments, including both Windows and Ubuntu applications, requires substantial engineering and annotation effort. It could become a useful resource if the environments and evaluation procedures are made very accessible.
-
It also has very useful empirical analysis beyond aggregate success rates, which includes repeated trials, domain-level results, workflow-length analysis, and procedural-guidance ablations. The analysis further illustrates concrete problems that arise during extended GUI execution.
Critical weaknesses
- One of my concern is the reliability of the benchmark's success labels is not sufficiently established. The benchmark relies heavily on model-based evaluation. According to Appendix D, 168 of the 338 tasks are evaluated through VLM-based inspection of the final GUI state, while another 102 use LLM-based artifact evaluation. Thus, approximately 80% of the tasks depend on model-based judgments.
- The paper positions Workflow-GYM as addressing a largely unexplored intersection of professional expertise, long-horizon execution, real GUI interaction, and economically meaningful outcomes. However, related benchmarks already investigate several important parts of this intersection such as OSWorld 2.0 and WeaveBench... I think the distinction from existing long-horizon professional workflow benchmarks needs stronger justification.
Questions for authors
- For the tasks using VLM/LLM-based verification, could the authors report human agreement on complete binary task outcomes rather than only individual criteria?
- How does Workflow-GYM differ substantively from recent long-horizon computer-use benchmarks?
- What are the concrete release and reproducibility plans for the benchmark?
2 comments
Question 3
We thank reviewer for raising this concern and agree the imortance of reproducibility of the results. To mitigate potential data contamination risks and preserve the reliability of benchmark evaluations, we have not yet publicly released the benchmark dataset. In the final version, we will publicly release a subset of Workflow-GYM, together with the corresponding execution environments and evaluation protocols, and report the performance of different models on this subset to facilitate independent evaluation and improve reproducibility.
We thank reviewer S4ry for providing insightful feedback. Below we provide point-by-point responses to the concerns raised:
Weakness 1 and Question 1
We thank reviewer for raising this important concern. Workflow-GYM emphasizes complex, end-to-end deliverables in real-world professional scenarios, where the required outcomes often involve visual semantics, spatial relationships, and domain-specific constraints that are difficult to comprehensively verify through deterministic rules. For example, one task requires agents to create a multi-layer 2D town map in Tiled, placing three types of characters in specific spatial relationships to a blue building (three to its left, three below it, and four inside it). Reliably verifying such requirements requires identifying visual entities and interpreting their spatial relationships, which are difficult to generalize through rule-based verification alone. Therefore, we adopt rule-based evaluation wherever feasible and employ model-based evaluators when necessary to verify expert-defined criteria.
Regarding evaluation reliability, as reported in Appendix D, we conducted a human–model agreement study on over 100 sampled evaluation cases, covering both VLM-based GUI-state evaluation and LLM-based artifact evaluation, achieving over 90% agreement. We clarify that this agreement is measured on binary task-level outcomes rather than individual criteria. Although exhaustive human verification of all task executions would be prohibitively expensive, our sampling-based validation provides empirical evidence supporting the reliability of model-based evaluation. We will clarify the agreement measurement and validation protocol in the revised version.
Weakness 2 and Question 2
We thank reviewer for raising this important point, and would like to clarify two aspects:
1. Contemporaneous work. Compared with Workflow-GYM, OSWorld 2.0[1] WeaveBench[2], were independently developed and publicly released around the same time in June 2026. According to the ICLR 2027 Reviewer Guidelines, authors are not required to compare against contemporaneous or unpublished arXiv works. Nevertheless, we appreciate the reviewer's suggestion and will incorporate a discussion of these related benchmarks in the revised version.
2. Distinct research objectives. Despite their shared focus on long-horizon computer use, these benchmarks address fundamentally different research questions. OSWorld 2.0 focuses on challenging interaction and reasoning phenomena in real-world computer-use workflows, while WeaveBench emphasizes cross-interface orchestration involving GUI, CLI, and code execution. In contrast, Workflow-GYM specifically investigates whether GUI agents can accomplish economically valuable, long-horizon workflows in real-world professional domains. Its systematic coverage of 338 tasks across 56 specialized software environments enables the evaluation of domain-specific knowledge, professional software proficiency, and long-horizon GUI execution in an integrated manner. We therefore view these benchmarks as complementary rather than redundant.
As shown in table below, although all 3 benchmarks investigate long-horizon computer use, they differ substantially in task selection principles, interaction paradigms, and targeted agent capabilities. Workflow-GYM uniquely emphasizes the systematic evaluation of economically valuable professional workflows through specialized GUI software.
References
[1]Yuan M, Zhou Z, Xiong X, et al. OSWorld2. 0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks[J]. arXiv preprint arXiv:2606.29537, 2026.
[2]Li W, Zhou B, Yu Y, et al. WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces[J]. arXiv preprint arXiv:2606.09426, 2026.