- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents SoftwareGym, an automated pipeline for synthesizing executable desktop software environments for CUA training. The authors construct 182 applications across twelve categories and introduce SoftwareGym-Bench, comprising 120 tasks with programmatic evaluation. Supervised fine-tuning of Qwen3.5-9B on trajectories from these applications improves Pass@3 from 41.39% to 56.97% on the evaluated 244-task OSWorld subset and from 25.8% to 40.0% on SoftwareGym-Bench. Ablations examine structured interface design, GUI-guided repair, and training trajectories from synthesized applications.
Critical strengths
- The paper provides a pipeline to generate executable applications with persistent state and multi-step workflows, covering a significant number and categories of applications.
- The potential of generalization to other software domains is encouraging. Training only on SoftwareGym trajectories improves the same base model by 15.58pp on the evaluated OSWorld subset, with gains across all eight reported application domains (Table 1). This supports the generalization of their synthesized environments beyond their own benchmark.
- The GUI-guided repair pipeline on synthesized applications is interesting. The ablation also shows its strength. Both branches start from the same applications and undergo two repair rounds, with GUI feedback withheld from the source-only branch. Across 36 applications, GUI-guided repair reaches 83.1%, compared with 48.8% for source-only repair.
Critical weaknesses
- SoftwareGym substantially overlaps with existing environment-synthesis pipelines. InfiniteWeb [1] combines functional and visual diversity, specification-driven generation, test-driven repair, and transfer to OSWorld. GUI-GENESIS [2] synthesizes executable applications with state-based rewards and static/dynamic verification. ScaleWoB [3] additionally performs interactive GUI validation followed by environment repair and revalidation, with human failure triage. The paper should clarify which mechanisms constitute substantive advances over these predecessors.
- Desktop application synthesis and explicit structural diversity controls are potentially valuable contributions. However, the current ablations compare against simplified design and source-only repair without establishing an advantage over closely related synthesis pipelines. A focused comparison under matched functionality and training budgets would help determine whether these components improve transfer beyond the benefits of additional synthetic supervision.
- The SFT is conducted only on Qwen3.5-9B, leaving it unclear whether the benefits of synthesized environments extend to larger models or other model families.
- In Table 3, combining OSWorld and SoftwareGym data reduces OSWorld Pass@3 from 61.48% to 58.61% relative to OSWorld-only training. Meanwhile, SoftwareGym-Bench performance increases from 40.0% to 47.5% relative to SoftwareGym-only. The results fail to demonstrate consistent complementarity across both benchmarks, which the author claims at Line 471. Further justification is required.
References [1] Zhang et al. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training. 2026. https://arxiv.org/abs/2601.04126 [2] Cao et al. GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training. 2026. https://arxiv.org/abs/2602.14093 [3] Liu et al. ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis. 2026. https://arxiv.org/abs/2605.25160
Questions for authors
- What are the substantive methodological advances over existing environment-synthesis pipelines? Please provide a mechanism-level comparison with previous works, distinguishing new components from adaptations of existing approaches. Which specific limitations of these predecessors does SoftwareGym address?
- Could the authors provide a focused comparison against a closely related synthesis approach, controlling functionality, trajectory quantity, and training budget? A smaller experiment on representative application categories would help isolate these components’ benefits from those of additional synthetic supervision.
- Could the authors validate the approach on at least one larger model or another model family? If additional training is impractical, please clarify the scope of the generality claim and discuss whether the observed gains might depend on the chosen backbone.
4 comments
Response to Reviewer Vbt1 (2/3)
Response to Weakness 2 and Question 2
... However, the current ablations compare against simplified design and source-only repair without establishing an advantage over closely related synthesis pipelines....Could the authors provide a focused comparison against a closely related synthesis approach, controlling functionality, trajectory quantity, and training budget? A smaller experiment on representative application categories would help isolate these components’ benefits from those of additional synthetic supervision.
We clarify that these methods target fundamentally different synthesis objectives. InfiniteWeb, GUI-GENESIS, and ScaleWoB focus on web-based environments or task-specific web surrogates, whereas SoftwareGym synthesizes complete, standalone desktop applications. Consequently, a direct comparison under matched functionality would require adapting these methods to a different environment substrate and synthesis objective.
To isolate the effect of interface diversity, we conducted a controlled Low-Diversity versus High-Diversity comparison. Low-Diversity contains 12 applications (one per category), while High-Diversity contains 182 applications across the same 12 categories. Both datasets were matched in size, category composition, difficulty, and quality, with identical training budgets.
Under matched training conditions, the results are shown in Table B. This demonstrates that greater software and interface diversity improves downstream transfer under the same data volume.
[Table B: Pass@1 comparison of Low-Diversity and High-Diversity training on the 244-task OSWorld application subset.]
| Domain | Low-Diversity | High-Diversity | Qwen3.5-9B Base |
|---|---|---|---|
| Chrome | 17/46 (36.96%) | 15/46 (32.61%) | 11/46 (23.91%) |
| GIMP | 17/26 (65.38%) | 17/26 (65.38%) | 10/26 (38.46%) |
| LibreOffice Calc | 4/47 (8.51%) | 10/47 (21.28%) | 6/47 (12.77%) |
| LibreOffice Impress | 13/47 (27.66%) | 10/47 (21.28%) | 13/47 (27.66%) |
| LibreOffice Writer | 7/23 (30.43%) | 11/23 (47.83%) | 9/23 (39.13%) |
| Thunderbird | 8/15 (53.33%) | 7/15 (46.67%) | 7/15 (46.67%) |
| VLC | 5/17 (29.41%) | 6/17 (35.29%) | 6/17 (35.29%) |
| VS Code | 11/23 (47.83%) | 14/23 (60.87%) | 6/23 (26.09%) |
| Overall | 82/244 (33.61%) | 90/244 (36.89%) | 68/244 (27.87%) |
Response to Weakness 3 and Question 3
The SFT is conducted only on Qwen3.5-9B, leaving it unclear whether the benefits of synthesized environments extend to larger models or other model families.....Could the authors validate the approach on at least one larger model or another model family? If additional training is impractical, please clarify the scope of the generality claim and discuss whether the observed gains might depend on the chosen backbone.
Following the reviewer's suggestion, we conducted an additional SFT experiment using Qwen3-VL-30B-A3B-Instruct, a substantially larger model from a different model family. We trained it using the same SoftwareGym training data as Qwen3.5-9B to examine whether the benefits of SoftwareGym generalize beyond the original backbone.
We evaluated the resulting model on the same 244 OSWorld application-domain tasks. The results are summarized in Table C:
The model achieves an overall average score of 48.32%, substantially exceeding the officially reported result of 30.60%. These results provide further evidence that the benefits of SoftwareGym training are not limited to Qwen3.5-9B and can extend to larger models from different model families.
[Table C: Performance of Qwen3-VL-30B-A3B-Instruct trained on SoftwareGym data across the eight OSWorld application domains.]
| Application Domain | Total Tasks | Pass | Pass Rate | Average Score |
|---|---|---|---|---|
| Chrome | 46 | 24 | 52.17% | 54.26% |
| GIMP | 26 | 17 | 65.38% | 65.38% |
| LibreOffice Calc | 47 | 14 | 29.79% | 29.79% |
| LibreOffice Impress | 47 | 18 | 38.30% | 40.34% |
| LibreOffice Writer | 23 | 12 | 52.17% | 56.51% |
| Thunderbird | 15 | 11 | 73.33% | 73.33% |
| VLC | 17 | 7 | 41.18% | 46.98% |
| VS Code | 23 | 11 | 47.83% | 47.83% |
| Overall | 244 | 114 | 46.72% | 48.32% |
Response to Reviewer Vbt1 (3/3)
Response to Weakness 4
In Table 3, combining OSWorld and SoftwareGym data reduces OSWorld Pass@3 from 61.48% to 58.61% relative to OSWorld-only training. Meanwhile, SoftwareGym-Bench performance increases from 40.0% to 47.5% relative to SoftwareGym-only. The results fail to demonstrate consistent complementarity across both benchmarks, which the author claims at Line 471. Further justification is required.
We clarify that mixed-source training should be assessed by overall performance across both benchmarks.
We have added the task-count-weighted Overall Pass@3 in Table D. The mixed model achieves the best overall result (54.95%), outperforming both OSWorld-only (52.19%) and SoftwareGym-only (51.38%) training, demonstrating complementary benefits from the two data sources.
[Table D: Pass@3 comparison of OSWorld-only, SoftwareGym-only, and combined training on OSWorld, SoftwareGym-Bench]
| Training source | OSWorld Pass@3 | SoftwareGym-Bench Pass@3 | Overall Pass@3 (364 tasks) |
|---|---|---|---|
| OSWorld-only | 61.48% | 33.3% | 52.19% |
| SoftwareGym-only | 56.97% | 40.0% | 51.38% |
| OSWorld + SoftwareGym | 58.61% | 47.5% | 54.95% |
We thank the reviewer again for these valuable suggestions. We believe the additional experiments and clarifications substantially strengthen our empirical evidence and address the concerns raised. We hope our responses address your concerns and would appreciate it if you could raise your rating.
Kind Reminder
Dear Reviewer Vbt1,
Thank you again for your thoughtful review. As the discussion period approaches its end, we would like to briefly follow up on our rebuttal.
We have provided additional experiments and clarifications addressing your concerns:
- Novelty and Comparison: We clarified SoftwareGym's distinction from existing environment-synthesis and CUA training methods, including comparisons with related approaches.
- Model Generalization: We conducted additional experiments with Qwen3-VL-30B-A3B, demonstrating the applicability of SoftwareGym to another model architecture.
- Data Diversity and Complementarity: We provided controlled diversity experiments and mixed-source training ablations to validate their contributions.
We would greatly appreciate your feedback on whether these results address your concerns and would be grateful if you could consider increasing your rating.
Thank you for your time and consideration.
The Authors
Response to Reviewer Vbt1 (1/3)
We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. In response, we conducted additional experiments and related work review. Our responses and additional results are detailed below.
Response to Weakness 1 and Question 1:
SoftwareGym differs from prior environment-synthesis pipelines in both the target environment substrate and the synthesis objective. While InfiniteWeb, GUI-GENESIS, and ScaleWoB primarily focus on web-based environments or task-specific web surrogates, SoftwareGym targets the synthesis of complete, runnable desktop applications for CUA training. The mechanism-level differences are summarized in Table A.
[Table A: Comparison of existing environment-synthesis methods across environment substrate, synthesis target, interaction scope, local OS/filesystem support, and validation mechanisms.]
The methodological contributions of SoftwareGym lie in three aspects:
Desktop application synthesis. We extend environment synthesis from web-based environments and task-specific surrogates to complete, runnable desktop applications with persistent local state and native and diverse GUI interactions.
Quantitative and iterative diversity control. SoftwareGym quantifies software and interface diversity and uses these measurements to guide successive synthesis iterations. This provides a systematic, measurement-driven approach to diversity control rather than relying solely on manually specified variations.
Closed-loop reliability assurance. We integrate functional specifications, source-level verification, GUI-based evaluation, and iterative repair to improve the functional correctness and reliability of generated applications.
Our contribution lies in integrating these mechanisms into a scalable pipeline for specification-driven, diversity-controlled desktop application synthesis.
Existing environment-synthesis approaches face two major limitations:
native-application-based approaches incur substantial integration costs, limiting software coverage and increasing the risk of application-specific overfitting.
browser-hosted environments offer scalable deployment and state control but are constrained by their shared browser/HTML/DOM substrate, which limits interaction depth and workflow complexity.
To address these limitations, SoftwareGym tackles three challenges:
Interface and interaction diversity. Unconstrained generation within the same software category often produces similar layouts and interaction patterns. SoftwareGym introduces quantitative structural-diversity measurement and iterative design refinement to generate distinct software structures and interfaces.
Functional reliability. Generated functions must remain usable throughout complete GUI workflows. SoftwareGym combines source-level verification, checkpoint-based GUI evaluation, and iterative repair to ensure functional correctness.
Scalability. Applying diversity controls and reliability checks across a large software pool creates substantial verification overhead. SoftwareGym automates the design–implementation–evaluation pipeline, enabling batch synthesis with minimal human intervention.