acceptodds
Under review as a conference paper at ICLR 2027

DeskForge: Dense Supervision From Desktop Environments for Computer-use agents

Abstract

Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. We will release the framework, dataset, and trained models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
computer-use agents, GUI grounding, visual grounding, vision-language models, multimodal learning, multimodal datasets, desktop environments, dense visual supervision, screen parsing, scalable data generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WJ7VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

DeskForge introduces a controllable Linux desktop environment and an automated pipeline for collecting GUI training data. It varies application states, content, window layouts, appearance, and resolution, and combines screenshots, accessibility trees, and window geometry to produce dense element annotations and before-and-after click transitions. The resulting DeskForge-1M dataset contains approximately 1.21 million desktop observations, 159.7 million element instances, and 917,000 click transitions. The authors assess its training utility through single-target grounding fine-tuning, multistep task execution under a fixed planner, and dense element detection.

Critical strengths
  1. DeskForge provides a reasonably comprehensive engineering implementation. It combines configurable multi-application desktop scenes with annotations of element hierarchy, window ownership, and visible regions under occlusion, providing useful infrastructure for studying grounding in complex desktop environments.
  2. The experiments demonstrate training benefits across models and datasets. All four models improve on all five external grounding benchmarks.
Critical weaknesses
  1. The contribution is limited relative to prior work. Existing GUI grounding efforts, including OS-Atlas [1] and ScaleCUA [2], also collect data through executable environments, interface structure extraction, automated exploration, and instruction synthesis. Although DeskForge emphasizes integrating multi-application composition, configurable appearance, occlusion geometry, and dense annotations into one system, ScaleCUA has already explored similar capabilities. Moreover, the datasets compared in Table 1 are from 2025 or earlier, leaving the comparison insufficiently current.
  2. The absence of data comparisons under matched training budgets makes it difficult to attribute the gains to DeskForge’s distinctive design. The main experiments compare models before and after fine-tuning, establishing only that additional grounding training is beneficial. The paper does not compare existing grounding datasets using the same backbone, sample count, and training budget, nor does it isolate the contributions of multi-window composition, occlusion, and appearance variation through ablations. The appendix’s breakdowns by scene difficulty do not substitute for these controls.
  3. The external benchmark comparisons are insufficient to establish competitiveness. Table 3 reports only the base and fine-tuned results of four models, without representative strong grounding baselines. This omission is particularly noticeable because Table 2 already includes several relevant strong models, yet their external benchmark performance is not reported. Recent models achieve substantially higher scores on these benchmarks, and some benchmarks arguably offer little remaining discriminatory value. Improvements over the starting checkpoints therefore do not establish the dataset’s incremental value for current leading models.
  4. In my view, the central grounding challenge for GUI agents in 2026 has shifted. The main bottleneck is no longer the lack of grounding data synthesis pipelines or poor performance in densely populated interfaces, but the ability to execute sequences of grounding actions quickly and accurately in tasks that require frequent, precise interaction. The GPT-6-Astra character-drawing demo illustrates this distinction well.

[1] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. International Conference on Learning Representations, 2025, pp. 5090–5108.

[2] Liu, Z., Xie, J. J., Ding, Z., et al. ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. International Conference on Learning Representations, 2026, pp. 62707–62766.

Questions for authors
  1. Relative to the closest automated collection and synthesis methods, which additional capability constitutes DeskForge’s core contribution? Can a direct comparison demonstrate that it addresses a gap that existing pipelines do not adequately cover?
  2. Under the same backbone, training sample count, and computational budget, how does training on DeskForge compare with training on existing public grounding data or a mixture of the two? Does adding DeskForge data yield further gains for strong grounding models that have already received extensive training?
  3. Could the authors report the performance of representative strong models from Table 2 on the external benchmarks in Table 3 and clarify differences in inference settings? How does the proposed approach’s absolute performance relative to these models support its practical value?
  4. Could controlled ablations establish the individual benefits of multi-application composition, occlusion variation, and appearance variation? Are there training experiments that directly use the dense structural annotations and click transitions to demonstrate their value beyond conventional single-target grounding supervision?
7 comments
AuthorsOpenReview ↗

Response to Reviewer WJ7V (1/3)

We thank the reviewer for the detailed assessment. We address the points in order.

W1 / Q1: the additional capability, and a direct comparison

The core contribution is controllable scene construction on real applications. DeskForge specifies the desktop scene before capture (the open applications, the layout and overlap of their windows, the theme and the resolution), executes it on running software, and records the visible region of every element.

  • Collection pipelines (OS-Atlas [1], ScaleCUA [2]) operate real software across platforms, which the revised Table 1 credits explicitly. They record the screen as exploration reaches it: ScaleCUA, for example, retains "only the topmost, unobscured elements" ([2], App. A.5.3) and excludes "elements from background applications" ([2], App. A.7.2).
  • Synthesis pipelines (WinDeskGround, GUI-Owl-1.5, AutoGUIWorld) control the screen but construct it from captured or generated images.
  • DeskForge controls the scene and executes it, retaining partially visible elements with their visible regions and the outcomes of executed clicks.
ResourceLive softwareScene parametersVisibility geometry
OS-Atlas✓not documented✗
ScaleCUA✓partial✗
WinDeskGround✗✓partial
DeskForge✓✓✓

Among the listed resources, DeskForge is the only one that provides both components and stores the exposed region of every element.

Direct comparison: targeted generation. Since the scene is specified before capture, DeskForge can generate supervision for a condition on which a model is weak, which existing pipelines do not cover. We replaced 25K of 200K training examples with examples whose targets are partially occluded, keeping the budget fixed.

Training dataWDG L1–L5WDG L4WDG L5
200K general65.147.620.8
175K general + 25K targeted68.954.427.8

Accuracy improves most on the levels with the heaviest occlusion (+6.8 on L4, +7.0 on L5), while the external mean is unchanged (68.7 vs. 68.8). The training scenes are Linux desktops, and the gain transfers to the Windows desktops of WinDeskGround (WDG). With these examples, the 4B model exceeds GUI-Owl-1.5-32B on this benchmark (68.9% vs. 65.9%). This form of improvement is not available from a fixed dataset; it follows from the controllability of the framework, which we regard as its central capability. At a matched budget, DeskForge data also outperform ScaleCUA data (W2 / Q2 below).

Currency of Table 1. The submitted table already included three resources from 2026 (ScreenParse, MolmoPoint-GUISyn, WinDeskGround); the revised table adds GUI-360, CUA-Suite, GUI-Owl-1.5 and AutoGUIWorld.

W2 / Q2: comparison under the same backbone, sample count and budget

We fine-tuned the same backbone (Qwen3.5-4B, rank-8 LoRA) on 200K examples from each source, with identical training settings and evaluation protocol. The table reports means over three training seeds; per-benchmark results are in Table 4 of the revised PDF.

Training data (200K)External mean (5 benchmarks)WinDeskGround
UGround66.73 ± 0.3763.33 ± 0.25
GroundCUA66.63 ± 0.2460.55 ± 0.84
ScaleCUA66.94 ± 0.7763.46 ± 0.23
DeskForge69.30 ± 0.4964.79 ± 0.37

DeskForge is the strongest source on both measures, so the gains are not explained by additional grounding training alone. Beyond this result, the contribution lies in how the data are obtained: generation is fully automated, requires no human annotation, scales with compute, and is controllable at the level of the scene, a combination that existing sources do not offer. Regarding models that have already received extensive grounding training: of the four models fine-tuned in the submission, UI-R1-3B is a GUI-grounding specialist, and it improves on all five external benchmarks and from 32.0 to 38.0 solved WebArena-Infinity tasks (five planner seeds).

[1] Wu, Z., Wu, Z., Xu, F., et al. OS-ATLAS: Foundation Action Model for Generalist GUI Agents. ICLR 2025.

[2] Liu, Z., Xie, J. J., Ding, Z., et al. ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer WJ7V (2/3)

W2 / Q4: individual benefits of composition, occlusion and appearance; use of dense annotations and transitions

To separate these factors, we conducted matched-size ablations in which only one factor varies. The results over three seeds are summarised below; the full tables are provided in Appendix E.3 (Tables 19 and 20).

FactorMatched comparisonWhere the factor appliesControl
Multi-application compositionsingle- → multi-application scenes (100K)WinDeskGround: 60.5 → 63.9external mean: 66.5 → 68.4
Visible-region labelsfull box → visible region (identical 100K frames)occluded targets: 52.5 → 55.4unoccluded targets: 77.5 → 77.5
Targeted occlusion25K general → 25K occluded-target examples (200K total)WinDeskGround L5: 20.8 → 27.8external mean: 68.7 → 68.8
Appearanceone preset → six presets (50K)external mean: 67.2 → 67.7WinDeskGround: 63.7 → 63.9

Composition is the largest factor and also improves the external benchmarks, which consist predominantly of single-application screenshots. The benefit of occlusion depends on the label and on control: full-box supervision is no better than excluding occluded targets (52.5 vs. 52.8), whereas visible-region labels and targeted generation both yield gains. Appearance has the smallest effect.

The dense annotations are used directly by the element detector in the paper, which transfers to GroundCUA and outperforms the evaluated OmniParser and ScreenParse detectors. Every single-target example is also derived from the dense structure: an unambiguous instruction requires knowledge of all elements on the screen, the click target is the visible region, and the recorded transition verifies that the click produces a state change. We do not claim that auxiliary dense-prediction or transition-prediction objectives further improve click accuracy; such objectives are typically applied at pre-training scale (e.g., UI-TARS [3]).

W3 / Q3: strong models on the external benchmarks; inference settings

As requested, we evaluated nine public models on the external benchmarks, each with its documented native prompt and preprocessing. The full table and the per-model inference settings are provided in Appendix C.3 (Table 12).

ModelSS-ProSS-v2OSWorld-GUI-VisionMMB-GUIMean
Qwen3.5-4B (base)38.1080.3052.3017.8064.3050.56
Qwen3.5-4B + DeskForge63.2093.7068.1039.8082.7069.50
GUI-Owl-1.5-32B71.4795.2069.8639.0287.2672.56
GroundNext-7B54.2190.3367.9160.7681.3970.92
ScaleCUA-7B47.4491.9864.3625.6077.8561.45
UI-TARS-1.5-7B42.9590.0265.4323.2574.3559.20
OS-Atlas-Pro-7B16.8979.0133.878.2950.4737.71

The DeskForge-trained 4B model exceeds seven of the nine public models, all of which are larger (7B to 32B), on all five benchmarks, and comes within 3.1 points of the mean of the 32B model. We agree that several of these benchmarks are close to saturation; we therefore add WinDeskGround, where substantial headroom remains. On its hardest level (L5), the best public model, UI-Venus-2-9B, reaches 32.3%, GUI-Owl-1.5-32B 21.5% and GroundNext-7B 17.8%; the DeskForge-trained model reaches 20.8%, and 27.8% with targeted examples.

Protocol. The submitted Table 3 used a single controlled protocol per base/fine-tuned pair (2.1M-pixel image cap, thinking disabled), since its purpose was to isolate the effect of the data. Here, public models are evaluated with their native settings, and for Qwen3.5-4B (base and fine-tuned) the image cap is raised to 8.3M pixels; thinking remains disabled. The higher absolute scores relative to the submission, most notably on ScreenSpot-Pro, result from this larger input resolution, as the benchmark consists of high-resolution screenshots. With thinking enabled, the base model obtains a mean of 65.2 with its native tool format (57.2 with our prompt format), which remains 4.3 points below the fine-tuned model. The submitted table is retained in Appendix E.1 (Table 16).

[3] Qin, Y., et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326, 2025.

AuthorsOpenReview ↗

Response to Reviewer WJ7V (3/3)

W4: where the grounding challenge lies

We share the view that the practical objective is the fast and accurate execution of action sequences. We respectfully disagree, however, that grounding data and densely populated interfaces are no longer bottlenecks. This assessment concerns the direction of the field rather than a claim or result of the submission, and we are not aware of a published evaluation of the cited demonstration, which concerns a proprietary system; we therefore base our response on published results from 2026 and on our own experiments. For open, compact action models, which are the low-latency option, both indicate that grounding remains a limiting factor:

  • Sequences. With the planner fixed, replacing only the action-model weights raises the number of solved tasks from 30.6 to 47.6 of 119 on WebArena-Infinity and from 1.8 to 13.2 of 100 on OpenApps (five paired seeds). Independently, OSWorld-Human [4, §3.4] reports that in 23% of an agent's failures the plan is correct and the predicted coordinates are wrong.
  • Dense interfaces. On the hardest WinDeskGround level the best public model reaches 32.3%, and the benchmark paper reports that "occlusion emerges as the most critical bottleneck for all models" [5, §4.2.1].
  • Data. Precise interaction is itself reported to be data-limited: on drag grounding, the gains from fine-tuning "come entirely from data" (DragOn [6, §5.3]).

We ask the reviewer to reconsider this point in light of the evidence above, and to indicate which of the remaining concerns most strongly determines the recommendation.

[4] Abhyankar, R., Qi, Q., Zhang, Y. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents. MLSys 2026.

[5] Zhao, H., Chen, T., Wang, Z. WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments. ICML 2026.

[6] Bout, N., Langevin, M., Riochet, R. DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions. ICML 2026 SCALE Workshop.

Reviewer_783zOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeskForge, a configurable desktop environment that composes and explores running Linux applications to collect dense GUI supervision. Its annotation pipeline combines accessibility information, screenshots, and window geometry to represent element structure, visible regions, text, and interaction outcomes. The resulting DeskForge-1M dataset contains approximately 1.2M observations, 159.7M elements, and 917K transitions. Fine-tuning four models on 200K grounding examples improves performance on internal held-out conditions and five external grounding benchmarks. The paper additionally demonstrates improved task completion under a fixed planner and cross-dataset element detection.

Critical strengths
  1. DeskForge combines real multi-application execution with dense, visibility-aware annotations. Handling overlapping windows and distinguishing accessible names from rendered text make it a useful resource for desktop grounding.
  2. The data demonstrate broad practical utility: all four models improve on five external benchmarks. Qwen gains 11.51 percentage points on ScreenSpot-Pro, while its task completions under a fixed planner increase from 31 to 50 on WebArena-Infinity.
Critical weaknesses
  1. The experiments lack matched comparisons against existing training datasets and ablations of composition, appearance variation, and instruction generation. They demonstrate the usefulness of additional DeskForge training but do not isolate the benefits of its distinctive design choices.

  2. Quality validation is incomplete. Element correctness does not establish annotation completeness, and the missing-element diagnostic covers only 29 previously inspected screens.The instruction audit covers training and validation examples and relies on only one reviewer. The quality of held-out evaluation instructions is not independently assessed in the reported audit.

Questions for authors
  1. How does DeskForge compare with existing grounding data under the same backbone, training settings, and trained sample count?

  2. Can you audit missing elements and held-out instruction–target pairs, including breakdowns by application and occlusion?

  3. Are the task-completion gains stable across repeated paired runs? (given the planner’s sampling temperature of 0.7 is used)

  4. Please clarify how “Configurable Environment” is defined in Table 1 and applied consistently across baselines. Does Table 1 overstate DeskForge’s advantages or misrepresent the capabilities of the baselines?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 783z (1/2)

We thank the reviewer for the constructive and precise feedback. Each of the four questions led to a new experiment or a clearer table in the revised paper; we answer them in order.

Q1 / W1: comparison with existing grounding data, and ablations of the design choices

To answer this question, we fine-tuned the same backbone (Qwen3.5-4B, rank-8 LoRA) on 200K examples from each source, with identical training settings and evaluation protocol. The table reports means over three training seeds.

Training data (200K)External mean (5 benchmarks)WinDeskGround
UGround66.73 ± 0.3763.33 ± 0.25
GroundCUA66.63 ± 0.2460.55 ± 0.84
ScaleCUA66.94 ± 0.7763.46 ± 0.23
DeskForge69.30 ± 0.4964.79 ± 0.37

At a matched budget, DeskForge obtains the highest mean over the five external benchmarks (2.4 to 2.7 points above the other sources) and the highest accuracy on WinDeskGround, an independent benchmark of multi-window desktops, while being generated fully automatically and without human annotation.

To isolate the individual design choices, we further conducted matched-size ablations in which only one factor varies. The results over three seeds are summarised below, and the full tables are provided in Appendix E.3 (Tables 19 and 20).

FactorMatched comparisonWhere the factor appliesControl
Multi-application compositionsingle- → multi-application scenes (100K)WinDeskGround: 60.5 → 63.9external mean: 66.5 → 68.4
Visible-region labelsfull box → visible region (identical 100K frames)occluded targets: 52.5 → 55.4unoccluded targets: 77.5 → 77.5
Appearanceone preset → six presets (50K)external mean: 67.2 → 67.7WinDeskGround: 63.7 → 63.9

Among the design choices, composition contributes most and also improves the external benchmarks, which consist predominantly of single-application screenshots; visibility-aware labels help specifically on occluded targets; appearance variation has the smallest effect, which we state in the revision. In addition, since scenes are specified before capture, examples for a difficult condition can be generated on request: replacing 25K of the 200K examples with partially occluded targets raises the hardest WinDeskGround level from 20.8% to 27.8% at a fixed budget. Instruction generation is assessed directly by the audit below.

Q2 / W2: audit of missing elements and held-out instruction–target pairs

We agree that element correctness does not establish completeness, and that the earlier diagnostic was limited in scope. We therefore conducted an additional human audit with two independent annotators who had no prior exposure to the samples. To assess completeness, they examined 30 previously uninspected screens, which cover all 19 applications and 4,537 annotated elements, with 10 screens per occlusion level. To assess instruction quality, they judged 150 held-out instruction–target pairs, balanced over the four held-out conditions and stratified by target occlusion.

Audit measureResult
Recall of visible elements (mean of annotators)99.2%
Recall by occlusion level: low / mid / high99.5 / 98.9 / 99.1%
Recall across the 19 applications91.8–100% (median 99.8%)
Held-out instructions that clearly and uniquely identify the target95.3%
By target visibility: fully visible / partly occluded97.6 / 94.5%
Accepted by both / one / neither annotator136 / 14 / 0

Recall is uniformly high across occlusion levels and applications. For the instructions, the two annotators agree on 90.7% of the judgements (Gwet's AC1 = 0.90), and no pair was rejected by both. The results by held-out condition and by application are provided in Appendix D.1 (Tables 14 and 15).

AuthorsOpenReview ↗

Response to Reviewer 783z (2/2)

Q3: stability of the task-completion gains across repeated paired runs

We repeated the evaluation with five planner seeds, keeping the planner and its sampling temperature unchanged and pairing seeds between the base and the fine-tuned action model. The table reports the number of solved tasks (mean ± s.d.).

Action modelWebArena-Infinity (119): base+ DeskForgeOpenApps (100): base+ DeskForge
Gemma4-E4B4.2 ± 0.840.4 ± 2.11.2 ± 1.38.4 ± 3.5
UI-R1-3B32.0 ± 1.638.0 ± 1.68.0 ± 1.611.6 ± 1.1
InternVL3.5-8B38.0 ± 1.644.4 ± 2.15.0 ± 1.610.4 ± 1.5
Qwen3.5-4B30.6 ± 1.147.6 ± 1.81.8 ± 0.813.2 ± 3.2

The gains are stable: every fine-tuned model solves more tasks than its base in each of the five paired seeds, in both environments. The revised paper reports these five-seed results in place of the single runs (Table 5).

Q4: definition of "Configurable Environment" in Table 1

By a configurable environment we refer to a pipeline in which the scene is specified before capture and realised on running software: the open applications, their state, the layout and overlap of their windows, the theme and the resolution. We agree that a single binary column left this ambiguous and could not credit partial support. The revised Table 1 therefore separates these two components and uses three levels of support. An excerpt is shown below.

ResourceLive softwareScene parametersVisibility geometry
OS-Atlas✓not documented✗
ScaleCUA✓partial✗
WinDeskGround✗✓partial
DeskForge✓✓✓

OS-Atlas and ScaleCUA are now credited for collecting from live software, and WinDeskGround for the parametric layout of captured window images. DeskForge remains the only listed resource that provides both components and stores the exposed region of every element. The table also adds four recent resources; definitions and the sources for each resource are in Appendix F.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 783z,

Thank you again for your constructive review. As the discussion period closes on 18 November, we would like to ask whether our response resolves your questions. Following your suggestions, we added the matched-budget comparison and ablations, a two-annotator audit of completeness and held-out instructions, five-seed task-completion results, and a revised Table 1.

We would be glad to clarify any remaining point before the deadline, and we would be grateful if you would consider these additions in your final assessment.

Sincerely,

The authors

AuthorsOpenReview ↗

Final follow-up before the discussion deadline

Dear Reviewer 783z,

As the author-reviewer discussion period closes today, 18 November, we would like to ask once more whether our response addresses your concerns. We remain available to clarify any remaining point before the deadline.

Thank you again for your time.

Sincerely,

The authors

Reviewer_Kq2eOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeskForge is a controllable desktop environment for generating supervision for computer-use agents. It composes real Linux applications under different window layouts, appearances, resolutions, and application states, and combines screenshots, accessibility trees, and window geometry to produce dense annotations and recorded click transitions. The resulting DeskForge-1M contains 1.21M observations and 159.7M element instances. Fine-tuning four vision-language models on 200K grounding examples improves performance on held-out desktop conditions and external benchmarks. Additional experiments evaluate downstream task completion under a fixed planner and dense element detection.

Critical strengths
  1. A substantial data-collection and annotation effort. The framework integrates real application execution, configurable desktop composition, and visibility-aware annotations. Retaining exposed element fragments, window ownership, and action-linked observations makes the resource potentially useful for several computer-use tasks.

  2. Consistent improvements over the starting checkpoints. All four fine-tuned models improve across the five external grounding benchmarks. The fixed-planner evaluation also demonstrates downstream benefits: Qwen3.5-4B increases from 31 to 50 solved WebArena-Infinity tasks and from 3 to 15 OpenApps tasks. These results establish that the generated data provides useful supervision.

Critical weaknesses
  1. The methodological contribution is limited and its distinctive components are not sufficiently validated. Accessibility-based collection, synthetic instruction generation, dense GUI annotation, and supervised grounding are established approaches. DeskForge’s main distinction is their integration within configurable multi-application scenes, together with visibility refinement. This is useful engineering, but the paper does not provide controlled ablations showing how much multi-window composition, appearance variation, or visibility-aware targets contribute. Moreover, the main VLM experiments use single-target supervision rather than directly learning the richer structure or transitions emphasized by the dataset. Without these analyses, the paper offers limited evidence that its particular design constitutes a substantial advance.

  2. The evaluation does not establish competitiveness or an advantage over existing datasets. Table 3 mainly compares fine-tuned models with their starting checkpoints, rather than strong specialist models under the same evaluation protocol. Published GroundNext results already provide stronger reference performance: GroundNext-3B with SFT reports 48.6% on ScreenSpot-Pro, compared with 41.68% for the strongest DeskForge-trained model in Table 3. These are not matched comparisons—GroundNext uses 700K SFT examples and a different training setup—but they show why improvements over base models alone are insufficient to establish competitiveness. Crucially, the paper does not train the same backbone on an equally sized GroundCUA subset using the same recipe. It therefore remains possible that existing data would deliver comparable or better gains, without the proposed collection pipeline. GroundCUA / GroundNext, Tables 2 and 18.

Questions for authors
  1. How does training on 200K DeskForge examples compare with 200K GroundCUA examples using the same backbone, optimization settings, image resolution, and evaluation protocol?

  2. What improvements can be attributed specifically to multi-application composition and visibility-aware annotations, beyond increasing the amount of grounding supervision? Matched-size ablations would help establish this.

  3. Can you include GroundNext and other strong grounding models in the external evaluation under a consistent protocol? GroundNext appears in the internal evaluation, but the external table does not establish how DeskForge-trained models compare against it.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Kq2e (1/2)

We thank the reviewer for the careful assessment and the specific experiments requested. We agree that improvements over the starting checkpoints cannot establish whether existing data would yield comparable gains, and we ran the requested comparisons.

Q1 / W2: 200K DeskForge vs. 200K GroundCUA under an identical setup

Following the reviewer's suggestion, we fine-tuned the same backbone (Qwen3.5-4B) on 200K examples from each source, using an identical recipe (rank-8 LoRA), image resolution and evaluation protocol. In addition to GroundCUA, we include UGround and ScaleCUA. The table reports means over three training seeds.

Training data (200K)SS-ProSS-v2OSWorld-GUI-VisionMMB-GUIMean (5)WinDeskGround
UGround58.2189.8067.4339.2378.9866.73 ± 0.3763.33 ± 0.25
GroundCUA60.1488.7065.7740.4178.1566.63 ± 0.2460.55 ± 0.84
ScaleCUA59.7388.8869.2638.4378.3966.94 ± 0.7763.46 ± 0.23
DeskForge62.8293.5068.5039.5082.1669.30 ± 0.4964.79 ± 0.37

At an equal budget, DeskForge obtains the highest mean over the five external benchmarks and the highest accuracy on WinDeskGround, an independent benchmark of multi-window Windows desktops; GroundCUA and ScaleCUA are each ahead on one benchmark. The gains in the paper are therefore not explained by additional grounding supervision alone.

We also wish to emphasise that this comparison evaluates only one property of the data. The main contribution of DeskForge is the framework itself: the examples are generated fully automatically, without the human annotation on which GroundCUA relies, at a scale and under scene conditions specified by the user, and with annotation quality verified by an independent audit (99.2% recall of visible elements). The ablations below show that this control is itself beneficial.

Q2 / W1: contributions of multi-application composition and visibility-aware targets

To isolate these two components, we conducted matched-size ablations in which only one factor varies, while the applications, themes, resolutions and instruction generator are held fixed. The table below summarises the results over three seeds; the full tables with standard deviations are provided in Appendix E.3 (Tables 19 and 20).

FactorMatched comparisonWhere the factor appliesControl
Multi-application compositionsingle- → multi-application scenes (100K)WinDeskGround: 60.5 → 63.9external mean: 66.5 → 68.4
Visible-region labelsfull box → visible region (identical 100K frames)occluded targets: 52.5 → 55.4unoccluded targets: 77.5 → 77.5
Appearanceone preset → six presets (50K)external mean: 67.2 → 67.7WinDeskGround: 63.7 → 63.9

Composition is the largest factor, and it also improves the external benchmarks, which consist predominantly of single-application screenshots. Visible-region supervision helps only where targets are occluded; full-box supervision is no better than excluding occluded targets (52.5 vs. 52.8). Appearance has the smallest effect.

Targeted generation. Since the scene is specified before capture, DeskForge can generate supervision for a condition on which a model is weak. To illustrate this, we replaced 25K of the 200K training examples with examples whose targets are partially occluded, keeping the budget fixed.

Training dataWDG L1–L5WDG L4WDG L5
200K general65.147.620.8
175K general + 25K targeted68.954.427.8

Accuracy improves most on the levels with the heaviest occlusion (+6.8 on L4, +7.0 on L5), while the external mean is unchanged (68.7 vs. 68.8). The training scenes are Linux desktops, and the gain transfers to the Windows desktops of WinDeskGround (WDG). With these examples, the 4B model exceeds GUI-Owl-1.5-32B on this benchmark (68.9% vs. 65.9%). This form of improvement is not available from a fixed dataset; it follows from the controllability of the framework, which we regard as its central capability.

AuthorsOpenReview ↗

Response to Reviewer Kq2e (2/2)

Q3 / W2: GroundNext and other strong models under a consistent protocol

We evaluated nine public models on the five external benchmarks, each with its documented native prompt and preprocessing (full table and per-model settings in Appendix C.3, Table 12). We use the released GroundNext-7B checkpoint, as the 3B checkpoint was not available to us.

ModelSS-ProSS-v2OSWorld-GUI-VisionMMB-GUIMean
Qwen3.5-4B (base)38.1080.3052.3017.8064.3050.56
Qwen3.5-4B + DeskForge63.2093.7068.1039.8082.7069.50
GUI-Owl-1.5-32B71.4795.2069.8639.0287.2672.56
GroundNext-7B54.2190.3367.9160.7681.3970.92
ScaleCUA-7B47.4491.9864.3625.6077.8561.45
UI-TARS-1.5-7B42.9590.0265.4323.2574.3559.20
UGround-V1-7B31.4489.7847.5214.0468.3150.22
OS-Atlas-Pro-7B16.8979.0133.878.2950.4737.71

On ScreenSpot-Pro, the benchmark cited in the review, the DeskForge-trained 4B model obtains 63.2%, against 54.2% for GroundNext-7B in our evaluation (52.9% as reported in [1]). It exceeds seven of the nine public models on all five benchmarks. GroundNext-7B retains a higher mean owing to UI-Vision; it is a 7B model trained, as the reviewer notes, on 700K examples, compared with a 4B model and 200K automatically generated examples here. On WinDeskGround, the DeskForge-trained model with targeted examples obtains 68.9%, against 63.4% for GroundNext-7B and 65.9% for GUI-Owl-1.5-32B (Appendix E.2, Table 18).

Protocol. The submitted Table 3 used a single controlled protocol per base/fine-tuned pair (2.1M-pixel image cap, thinking disabled), since its purpose was to isolate the effect of the data. Here, public models are evaluated with their native settings, and for Qwen3.5-4B (base and fine-tuned) the image cap is raised to 8.3M pixels; thinking remains disabled. The higher absolute scores relative to the submission, most notably on ScreenSpot-Pro, result from this larger input resolution, as the benchmark consists of high-resolution screenshots. With thinking enabled, the base model obtains a mean of 65.2 with its native tool format (57.2 with our prompt format), which remains 4.3 points below the fine-tuned model. The submitted table is retained in Appendix E.1 (Table 16).

W1: single-target supervision in the VLM experiments

The dense annotations are used directly by the element detector in the paper, which transfers to GroundCUA and outperforms the evaluated OmniParser and ScreenParse detectors on all three localisation metrics. Every single-target example is also derived from the dense structure: an unambiguous instruction requires knowledge of all elements on the screen, the click target is the visible region of the element, and the recorded transition verifies that the click produces a state change. We do not claim that auxiliary dense-prediction or transition-prediction objectives further improve click accuracy; such objectives are typically applied at pre-training scale (e.g., UI-TARS [2]), and the released annotations make this study possible.

[1] Feizi, A., Nayak, S., Jian, X., et al. Grounding Computer Use Agents on Human Demonstrations. ICLR 2026.

[2] Qin, Y., et al. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326, 2025.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Kq2e,

Thank you again for your careful review. As the discussion period closes on 18 November, we would like to ask whether our response addresses your concerns. Following your three questions, we added the matched-budget comparison with GroundCUA, the matched-size ablations of composition and visibility-aware targets, and the evaluation of GroundNext and other strong public models on the external benchmarks.

We would be glad to clarify any remaining point. If the concerns are resolved, we would be grateful if you would consider updating your assessment.

Sincerely,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive feedback. We are encouraged that all three reviewers consider DeskForge a useful resource for desktop grounding and note its consistent improvements across models and benchmarks (Kq2e, 783z, WJ7V).

The reviews share three requests: a comparison with existing data at a matched budget, ablations of the design choices, and a comparison with strong public models. We conducted all three, together with an independent annotation audit and repeated task-completion runs. The results are summarised below; tables and details are given in the individual responses and in the revised PDF.

Contribution. DeskForge is a framework rather than a fixed dataset: desktop scenes are specified before capture and executed on real applications, and dense annotations, including the visible region of every element, are produced fully automatically and without human annotation. The new experiments indicate that data generated in this way are competitive with existing sources, and that the control over the scene is itself beneficial.

Summary of new results

  1. Matched-budget comparison (Kq2e Q1, 783z Q1, WJ7V Q2). With the same backbone, recipe and 200K examples per source, DeskForge obtains the highest mean over the five external benchmarks (69.3, against 66.6–66.9 for UGround, GroundCUA and ScaleCUA) and the highest accuracy on WinDeskGround, an independent benchmark of multi-window desktops.
  2. Ablations (Kq2e Q2, 783z W1, WJ7V Q4). At matched size, multi-application composition is the largest factor (WinDeskGround 60.5 → 63.9; external mean 66.5 → 68.4). Supervision on the visible region improves accuracy on occluded targets (52.5 → 55.4) without affecting unoccluded ones. Appearance variation has the smallest effect.
  3. Targeted generation. Since scenes are specified before capture, supervision for a condition on which a model is weak can be generated on request. Replacing 25K of 200K training examples with partially occluded targets raises accuracy on the hardest WinDeskGround level from 20.8% to 27.8% at a fixed budget, with the external mean unchanged. This form of improvement is not available from a fixed dataset.
  4. Strong public models (Kq2e Q3, WJ7V Q3). We evaluated nine public models with their native settings, all of which are larger (7B to 32B). The DeskForge-trained 4B model exceeds seven of them on all five external benchmarks, and obtains 63.2% on ScreenSpot-Pro against 54.2% for GroundNext-7B.
  5. Stability of task completion (783z Q3). Over five paired planner seeds, all four fine-tuned action models solve more tasks than their base models in every seed (Qwen3.5-4B: 30.6 → 47.6 of 119 tasks on WebArena-Infinity; 1.8 → 13.2 of 100 on OpenApps).
  6. Annotation audit (783z Q2). Two independent annotators find that 99.2% of visible elements are annotated and that 95.3% of held-out instructions clearly and uniquely identify their target.
  7. Table 1 (783z Q4, WJ7V W1). The definition of a configurable environment is stated explicitly and separated into two components with three-level marks, and four recent resources are added.

Evaluation protocol. The submitted Table 3 used a controlled protocol with a 2.1M-pixel image cap, since its purpose was to isolate the effect of the data. For the comparison with public models, the cap is raised to 8.3M pixels, which accounts for the higher absolute scores, most notably on ScreenSpot-Pro. The submitted table is retained in Appendix E.1.

Revisions to the PDF. All changes are highlighted in blue. The main ones are the following:

  • We added the matched-budget comparison (Section 5.3, Table 4) and the ablation tables (Appendix E.3).
  • We added public models to the external comparison (Table 3) and list the inference settings of every model (Appendix C.3).
  • We replaced the single task-completion runs with results over five seeds (Section 5.4, Table 5).
  • We report the two-annotator audit, with results per application and per occlusion level (Appendix D.1).
  • We revised Table 1 and provide the definitions and the sources for each resource (Appendix F).

We would be glad to provide further clarification during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.