- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
DeskForge introduces a controllable Linux desktop environment and an automated pipeline for collecting GUI training data. It varies application states, content, window layouts, appearance, and resolution, and combines screenshots, accessibility trees, and window geometry to produce dense element annotations and before-and-after click transitions. The resulting DeskForge-1M dataset contains approximately 1.21 million desktop observations, 159.7 million element instances, and 917,000 click transitions. The authors assess its training utility through single-target grounding fine-tuning, multistep task execution under a fixed planner, and dense element detection.
Critical strengths
- DeskForge provides a reasonably comprehensive engineering implementation. It combines configurable multi-application desktop scenes with annotations of element hierarchy, window ownership, and visible regions under occlusion, providing useful infrastructure for studying grounding in complex desktop environments.
- The experiments demonstrate training benefits across models and datasets. All four models improve on all five external grounding benchmarks.
Critical weaknesses
- The contribution is limited relative to prior work. Existing GUI grounding efforts, including OS-Atlas [1] and ScaleCUA [2], also collect data through executable environments, interface structure extraction, automated exploration, and instruction synthesis. Although DeskForge emphasizes integrating multi-application composition, configurable appearance, occlusion geometry, and dense annotations into one system, ScaleCUA has already explored similar capabilities. Moreover, the datasets compared in Table 1 are from 2025 or earlier, leaving the comparison insufficiently current.
- The absence of data comparisons under matched training budgets makes it difficult to attribute the gains to DeskForge’s distinctive design. The main experiments compare models before and after fine-tuning, establishing only that additional grounding training is beneficial. The paper does not compare existing grounding datasets using the same backbone, sample count, and training budget, nor does it isolate the contributions of multi-window composition, occlusion, and appearance variation through ablations. The appendix’s breakdowns by scene difficulty do not substitute for these controls.
- The external benchmark comparisons are insufficient to establish competitiveness. Table 3 reports only the base and fine-tuned results of four models, without representative strong grounding baselines. This omission is particularly noticeable because Table 2 already includes several relevant strong models, yet their external benchmark performance is not reported. Recent models achieve substantially higher scores on these benchmarks, and some benchmarks arguably offer little remaining discriminatory value. Improvements over the starting checkpoints therefore do not establish the dataset’s incremental value for current leading models.
- In my view, the central grounding challenge for GUI agents in 2026 has shifted. The main bottleneck is no longer the lack of grounding data synthesis pipelines or poor performance in densely populated interfaces, but the ability to execute sequences of grounding actions quickly and accurately in tasks that require frequent, precise interaction. The GPT-6-Astra character-drawing demo illustrates this distinction well.
[1] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. International Conference on Learning Representations, 2025, pp. 5090–5108.
[2] Liu, Z., Xie, J. J., Ding, Z., et al. ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. International Conference on Learning Representations, 2026, pp. 62707–62766.
Questions for authors
- Relative to the closest automated collection and synthesis methods, which additional capability constitutes DeskForge’s core contribution? Can a direct comparison demonstrate that it addresses a gap that existing pipelines do not adequately cover?
- Under the same backbone, training sample count, and computational budget, how does training on DeskForge compare with training on existing public grounding data or a mixture of the two? Does adding DeskForge data yield further gains for strong grounding models that have already received extensive training?
- Could the authors report the performance of representative strong models from Table 2 on the external benchmarks in Table 3 and clarify differences in inference settings? How does the proposed approach’s absolute performance relative to these models support its practical value?
- Could controlled ablations establish the individual benefits of multi-application composition, occlusion variation, and appearance variation? Are there training experiments that directly use the dense structural annotations and click transitions to demonstrate their value beyond conventional single-target grounding supervision?
7 comments
Response to Reviewer WJ7V (2/3)
W2 / Q4: individual benefits of composition, occlusion and appearance; use of dense annotations and transitions
To separate these factors, we conducted matched-size ablations in which only one factor varies. The results over three seeds are summarised below; the full tables are provided in Appendix E.3 (Tables 19 and 20).
| Factor | Matched comparison | Where the factor applies | Control |
|---|---|---|---|
| Multi-application composition | single- → multi-application scenes (100K) | WinDeskGround: 60.5 → 63.9 | external mean: 66.5 → 68.4 |
| Visible-region labels | full box → visible region (identical 100K frames) | occluded targets: 52.5 → 55.4 | unoccluded targets: 77.5 → 77.5 |
| Targeted occlusion | 25K general → 25K occluded-target examples (200K total) | WinDeskGround L5: 20.8 → 27.8 | external mean: 68.7 → 68.8 |
| Appearance | one preset → six presets (50K) | external mean: 67.2 → 67.7 | WinDeskGround: 63.7 → 63.9 |
Composition is the largest factor and also improves the external benchmarks, which consist predominantly of single-application screenshots. The benefit of occlusion depends on the label and on control: full-box supervision is no better than excluding occluded targets (52.5 vs. 52.8), whereas visible-region labels and targeted generation both yield gains. Appearance has the smallest effect.
The dense annotations are used directly by the element detector in the paper, which transfers to GroundCUA and outperforms the evaluated OmniParser and ScreenParse detectors. Every single-target example is also derived from the dense structure: an unambiguous instruction requires knowledge of all elements on the screen, the click target is the visible region, and the recorded transition verifies that the click produces a state change. We do not claim that auxiliary dense-prediction or transition-prediction objectives further improve click accuracy; such objectives are typically applied at pre-training scale (e.g., UI-TARS [3]).
W3 / Q3: strong models on the external benchmarks; inference settings
As requested, we evaluated nine public models on the external benchmarks, each with its documented native prompt and preprocessing. The full table and the per-model inference settings are provided in Appendix C.3 (Table 12).
| Model | SS-Pro | SS-v2 | OSWorld-G | UI-Vision | MMB-GUI | Mean |
|---|---|---|---|---|---|---|
| Qwen3.5-4B (base) | 38.10 | 80.30 | 52.30 | 17.80 | 64.30 | 50.56 |
| Qwen3.5-4B + DeskForge | 63.20 | 93.70 | 68.10 | 39.80 | 82.70 | 69.50 |
| GUI-Owl-1.5-32B | 71.47 | 95.20 | 69.86 | 39.02 | 87.26 | 72.56 |
| GroundNext-7B | 54.21 | 90.33 | 67.91 | 60.76 | 81.39 | 70.92 |
| ScaleCUA-7B | 47.44 | 91.98 | 64.36 | 25.60 | 77.85 | 61.45 |
| UI-TARS-1.5-7B | 42.95 | 90.02 | 65.43 | 23.25 | 74.35 | 59.20 |
| OS-Atlas-Pro-7B | 16.89 | 79.01 | 33.87 | 8.29 | 50.47 | 37.71 |
The DeskForge-trained 4B model exceeds seven of the nine public models, all of which are larger (7B to 32B), on all five benchmarks, and comes within 3.1 points of the mean of the 32B model. We agree that several of these benchmarks are close to saturation; we therefore add WinDeskGround, where substantial headroom remains. On its hardest level (L5), the best public model, UI-Venus-2-9B, reaches 32.3%, GUI-Owl-1.5-32B 21.5% and GroundNext-7B 17.8%; the DeskForge-trained model reaches 20.8%, and 27.8% with targeted examples.
Protocol. The submitted Table 3 used a single controlled protocol per base/fine-tuned pair (2.1M-pixel image cap, thinking disabled), since its purpose was to isolate the effect of the data. Here, public models are evaluated with their native settings, and for Qwen3.5-4B (base and fine-tuned) the image cap is raised to 8.3M pixels; thinking remains disabled. The higher absolute scores relative to the submission, most notably on ScreenSpot-Pro, result from this larger input resolution, as the benchmark consists of high-resolution screenshots. With thinking enabled, the base model obtains a mean of 65.2 with its native tool format (57.2 with our prompt format), which remains 4.3 points below the fine-tuned model. The submitted table is retained in Appendix E.1 (Table 16).
[3] Qin, Y., et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326, 2025.
Response to Reviewer WJ7V (3/3)
W4: where the grounding challenge lies
We share the view that the practical objective is the fast and accurate execution of action sequences. We respectfully disagree, however, that grounding data and densely populated interfaces are no longer bottlenecks. This assessment concerns the direction of the field rather than a claim or result of the submission, and we are not aware of a published evaluation of the cited demonstration, which concerns a proprietary system; we therefore base our response on published results from 2026 and on our own experiments. For open, compact action models, which are the low-latency option, both indicate that grounding remains a limiting factor:
- Sequences. With the planner fixed, replacing only the action-model weights raises the number of solved tasks from 30.6 to 47.6 of 119 on WebArena-Infinity and from 1.8 to 13.2 of 100 on OpenApps (five paired seeds). Independently, OSWorld-Human [4, §3.4] reports that in 23% of an agent's failures the plan is correct and the predicted coordinates are wrong.
- Dense interfaces. On the hardest WinDeskGround level the best public model reaches 32.3%, and the benchmark paper reports that "occlusion emerges as the most critical bottleneck for all models" [5, §4.2.1].
- Data. Precise interaction is itself reported to be data-limited: on drag grounding, the gains from fine-tuning "come entirely from data" (DragOn [6, §5.3]).
We ask the reviewer to reconsider this point in light of the evidence above, and to indicate which of the remaining concerns most strongly determines the recommendation.
[4] Abhyankar, R., Qi, Q., Zhang, Y. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents. MLSys 2026.
[5] Zhao, H., Chen, T., Wang, Z. WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments. ICML 2026.
[6] Bout, N., Langevin, M., Riochet, R. DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions. ICML 2026 SCALE Workshop.
Response to Reviewer WJ7V (1/3)
We thank the reviewer for the detailed assessment. We address the points in order.
W1 / Q1: the additional capability, and a direct comparison
The core contribution is controllable scene construction on real applications. DeskForge specifies the desktop scene before capture (the open applications, the layout and overlap of their windows, the theme and the resolution), executes it on running software, and records the visible region of every element.
Among the listed resources, DeskForge is the only one that provides both components and stores the exposed region of every element.
Direct comparison: targeted generation. Since the scene is specified before capture, DeskForge can generate supervision for a condition on which a model is weak, which existing pipelines do not cover. We replaced 25K of 200K training examples with examples whose targets are partially occluded, keeping the budget fixed.
Accuracy improves most on the levels with the heaviest occlusion (+6.8 on L4, +7.0 on L5), while the external mean is unchanged (68.7 vs. 68.8). The training scenes are Linux desktops, and the gain transfers to the Windows desktops of WinDeskGround (WDG). With these examples, the 4B model exceeds GUI-Owl-1.5-32B on this benchmark (68.9% vs. 65.9%). This form of improvement is not available from a fixed dataset; it follows from the controllability of the framework, which we regard as its central capability. At a matched budget, DeskForge data also outperform ScaleCUA data (W2 / Q2 below).
Currency of Table 1. The submitted table already included three resources from 2026 (ScreenParse, MolmoPoint-GUISyn, WinDeskGround); the revised table adds GUI-360, CUA-Suite, GUI-Owl-1.5 and AutoGUIWorld.
W2 / Q2: comparison under the same backbone, sample count and budget
We fine-tuned the same backbone (Qwen3.5-4B, rank-8 LoRA) on 200K examples from each source, with identical training settings and evaluation protocol. The table reports means over three training seeds; per-benchmark results are in Table 4 of the revised PDF.
DeskForge is the strongest source on both measures, so the gains are not explained by additional grounding training alone. Beyond this result, the contribution lies in how the data are obtained: generation is fully automated, requires no human annotation, scales with compute, and is controllable at the level of the scene, a combination that existing sources do not offer. Regarding models that have already received extensive grounding training: of the four models fine-tuned in the submission, UI-R1-3B is a GUI-grounding specialist, and it improves on all five external benchmarks and from 32.0 to 38.0 solved WebArena-Infinity tasks (five planner seeds).
[1] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. ICLR 2025.
[2] Liu, Z., Xie, J. J., Ding, Z., et al. ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. ICLR 2026.