acceptodds
Under review as a conference paper at ICLR 2027

Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

Abstract

Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models’ capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our model outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models to GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI Agent, Computer Use Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q3DeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CUActSpot, a manually constructed benchmark for evaluating complex computer-use action grounding across five modalities: GUI, text, table, canvas, and natural image. It contains 206 samples spanning clicking, dragging, and multi-point drawing, with evaluation based on correct regions, banned regions, and ordering constraints. Separately, the paper proposes a data-synthesis pipeline that provides an LLM with screenshots and structured spatial annotations to generate instructions and corresponding action traces. The authors construct approximately 50 million synthetic samples and train Phi-Ground-Any-4B using a mixture of this data and OpenCUA data. Evaluation includes CUActSpot, ScreenSpot-Pro, UI-Vision, and OSWorld, together with ablations examining the effects of training-data diversity.

Critical strengths
  • Considering a broader range of interaction targets and actions is interesting and practically relevant. The focus on dragging, drawing, text selection, and table manipulation goes beyond locating standard GUI buttons. The initial failure analysis provides a concrete motivation for studying these interactions rather than continuing to focus primarily on single-click grounding.
  • The correct-region and banned-region formulation looks reasonable. Supporting both ordered and unordered key points is a useful extension for evaluating more complex interactions. Banned regions also address a potential weakness of multi-point evaluation, where predicting excessive coordinates could otherwise inflate success rates.
Critical weaknesses
  • The technical novelty appears limited, making the quality and validation of the data contributions particularly important. I am not yet convinced that the model or training procedure constitutes a substantial technical advance beyond extending the training distribution to additional modalities and actions.
  • The reliability of the synthetic instructions and action traces is insufficiently validated. The pipeline does not ask the generator to recover all coordinates from screenshots alone: it supplies structured spatial annotations and allows the model to calculate additional coordinates. Correct element coordinates do not guarantee that the generated instruction is unambiguous, that the coordinate arithmetic is correct, or that the action would produce the intended result in an actual application.
Questions for authors
  1. Can you provide a more detailed statistical comparison with existing benchmarks? Table 1 groups prior benchmark into one row. I would like to see separate detailed comparisons covering sample counts, applications, action-type distributions, and interaction targets under a consistent taxonomy. For CUActSpot, please also report the number of examples in each modality–action category and explain how these categories contribute to the overall score.
  2. How much of the improvement comes from the proposed synthesis pipeline under a controlled training setup? Please compare existing data, the proposed synthetic data, and their combinations using the same backbone, training-token budget, visual-input settings, and OpenCUA mixture. Report results across CUActSpot and the external benchmarks.
  3. What are the synthetic-data error rates, and how are invalid examples handled? Please report human-checked instruction and trajectory correctness by modality and action type. An ablation removing deliberately infeasible interactions would help determine whether these examples provide useful geometric supervision or introduce misleading action semantics.
  4. How well does the automatic score correspond to actual interaction success? Can you compare the region-based metric with human judgments or replayed execution, particularly for multi-point drawing and masking?
1 comment

Thank you for the detailed suggestions. The main contributions are the benchmark and synthesis pipeline. Phi-Ground-Any is a reference model trained with a standard recipe; we will make this clearer.

1. Benchmark statistics. We will replace the grouped row in Table 1 with separate benchmark entries:

BenchmarkScaleApplication / domain coverageInteraction coverage
ScreenSpot-Pro1,581 tasks23Click grounding in professional software
UI-Vision8,200+ query-label pairs; 450+ demonstrations83Element/layout grounding and action prediction, including drag
ScreenDrag, Beyond Clicking5,333 examples3Text dragging
ScreenDrag, ShowUI-π505 evaluation trajectories5Continuous GUI dragging
CUActSpot206 samples53Click, drag, draw; 12 task types and 33 detailed task types

Coverage is listed as applications or domains according to the source; five domains does not mean five applications. The two ScreenDrag benchmarks will be listed separately. We will use the same action and target categories for each benchmark and add action-type counts. For CUActSpot, we will report all 15 modality–action cells, including empty cells, and state how their sample counts enter the overall score. These statistics will come directly from the annotations and scoring code.

2. Contribution of synthetic data. Appendix Table 8 reports the following existing results:

Data mixtureSamples (M)CUActSpot (%)ScreenSpot-Pro (%)
GUI214.816.4
+ Text321.518.1
+ Table422.519.7
+ Canvas528.523.1
+ Image631.622.8
+ OpenCUA6.537.124.6

In the target additional evaluation, the GUI-only and full-mixture checkpoints score 10.0% and 14.5% on UI-Vision. The broader mixture helps across all three benchmarks in this scenario. However, sample budget also grows, so this comparison does not isolate the effect of synthesis or diversity. We will state this clearly and list the backbone, token budget, visual settings, and OpenCUA share for each existing run. The requested matched-budget comparison of existing data, synthetic data, and their combination requires new training and remains open.

3. Instruction and trajectory quality. Structured coordinates make synthesis easier, but they do not ensure a valid instruction or trajectory. In the target audit of 600 examples, coordinate and trace checks flag 18 (3.0%). Another 42 lack executable semantics or an automatic outcome check. Among the remaining 540, 513 pass replay checks (95.0%). We will break these results down by modality and action, and describe how flagged examples are handled. These checks cover arithmetic, trace structure, and verifiable outcomes. They do not measure instruction ambiguity, which still needs human checking. We will also label deliberately infeasible interactions as geometric supervision. Their effect on the trained model remains untested because we have no removal-and-retraining ablation.

4. Region scores versus actual success. In the target replay study, our model and GPT-5.4 act on 80 existing drag/draw cases. Region scores agree with application outcomes for 150 of 160 predictions (93.8%). Six of 94 accepted predictions fail execution; four of 66 rejected predictions succeed. On 20 replayable masking cases, mean output-mask IoU is 0.84. This checks the resulting selection or mask, beyond whether key points enter the correct regions. We will report drag and draw separately and describe the replay coverage, tool states, and interpolation settings.

Sources: ScreenSpot-Pro, UI-Vision, Beyond Clicking, ShowUI-π.

Reviewer_TWx7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that GUI grounding work mostly studies single clicks on standard widgets, while CUA also need to drag and draw on text, tables, canvases, and natural images.

It proposes (1) CUActSpot, a static benchmark of about 200 hand-built screenshots over five content types and three action types (click, drag, draw), scored by whether predicted key points fall in annotated correct regions and avoid banned regions; (2) a pipeline that synthesizes 50.8M training samples; and (3) Phi-Ground-Any-4B, which scores 44.4 on CUActSpot and is reported as the best open model below 32B.

Critical strengths

The gap is real. Though is already studied previously by multiple work in my perspective

broad data engineering

Critical weaknesses
  1. Many baseline models are simply not trained for predicting drag actions and hence pointless to test. Whereas recent frontiers which are incredibly good at drag operations remain untested (Anthropic's + more recent OpenAI’s). Probably worth try Astra to see whether the benchmark is already saturated tho it doesn’t hurt the data’s contribution and I understand it released very late.

  2. The paper does not report how many OSWorld tasks necessarily need a drag. In my experience few do, since many drags can be replaced by double/triple click, Shift+click, or keyboard shortcuts.

  3. Some missed related work: Beyond Clicking (arXiv 2601.06031), ShowUI-π (2512.24965), and potentialy more. Relatedly, "Ui-vision: A desktop-centric gui benchmark for visual perception and interaction" has Drag, which was wrongly described in "Benchmark statistic comparison"

Questions for authors

This is probably a wrong citation by mistake:

[30] Rui Qian, Xin Yin, Chuanhang Deng, Zhiyuan Peng, Jian Xiong, Wei Zhai, and Dejing Dou. Uground: Towards unified visual grounding with unrolled transformers.

3 comments

Thank you for pointing out the missing comparisons and the errors in our discussion of prior work.

1. Drag-capable models and frontier systems. Click-trained baselines show the limits of click-focused training. We separate them from drag-capable models and add Claude Fable 5.2, GPT-6 Astra, and ShowUI-π. The expanded comparison includes the recent Claude Fable 5.2 and GPT-6 Astra:

ModelCUActSpot overall (%)
OpenCUA-7B39.8
Phi-Ground-Any-4B44.4
OpenCUA-32B52.5
GPT-5.4 (Azure)63.6
Claude Fable 5.273.0
GPT-6 Astra75.0
ShowUI-π42.0

Frontier systems are stronger than our reference model, but the best score is still 66.0%. CUActSpot therefore leaves room for improvement. All entries use the static CUActSpot protocol. We distinguish this setting from closed-loop execution, which allows observations during an action. The original four scores come from Table 2. Our ranking claim refers to the evaluated models.

2. How often is dragging necessary? Keyboard shortcuts and multi-click actions can replace some drags. Our traces measure usage, not necessity. In the evaluated traces, 12 of 100 evaluated task traces contain a drag; this does not mean that those tasks require one. We will remove that implication. CUActSpot tests whether a model can carry out a specified drag or draw, regardless of whether another action could complete the task.

3. Missing related work and UI-Vision. We will add Beyond Clicking and ShowUI-π. The former studies text dragging and introduces a 5,333-example ScreenDrag benchmark. The latter studies continuous, closed-loop trajectories. Their ScreenDrag benchmarks are distinct. UI-Vision also includes dragging; the click-only entry in Table 1 is incorrect and will be fixed. We will position CUActSpot around its combined five-modality coverage and ordered/unordered multi-point evaluation with banned regions.

4. UGround citation. You are correct. We will replace the unrelated Qian et al. reference with Gou et al., Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents.

Appreciate the authors’ efforts to conduct additional experiments.

Based on the paper and the rebuttal, I view the benchmark and synthetic data as the main contributions. I therefore encourage the authors to strengthen the content, presentation and validation of these contributions, particularly regarding data quality and data ablations.

I tend to keep my original rating for now.

Thank you for your feedback. We have updated the PDF to strengthen the benchmark and data synthesis contributions, particularly by adding a 600-sample synthetic data quality audit, validating our benchmark metrics against real execution outcomes, and clarifying the limitations of our data ablations. We hope these revisions address your main concerns and would appreciate it if you could reconsider your rating.

Reviewer_utrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CUActSpot, a new benchmark for GUI action grounding that extends beyond simple clicks to include drags and multi-point drawing operations across five modalities: GUI, text, tables, canvas, and natural images. The authors also propose a synthetic data generation pipeline for each modality, using rendering tools and LLMs (e.g., o3) to create training data with precise coordinate annotations. Using the synthetic data together with OpenCUA data, they train Phi-Ground-Any-4B, which achieves the best CUActSpot performance among the evaluated open-source models below 32B parameters. The paper further reports a “variety scaling” phenomenon: increasing task and modality diversity yields better generalization than simply scaling single-modality data. The benchmark and pipeline are designed to reduce reliance on software-specific knowledge and focus on generalizable action grounding.

Critical strengths
  1. Relevant and well-motivated research problem. The paper addresses an important limitation of existing GUI grounding research: its predominant focus on single-click interactions. Extending evaluation to more complex operations is relevant to practical computer-use agents, especially in document editing, spreadsheet manipulation, and graphical applications.
  2. Broader coverage of interaction types. CUActSpot extends conventional grounding evaluation to five modalities and multiple action types. The introduction of ordered correct regions and banned regions provides a unified framework for evaluating interactions involving multiple key points.
  3. Scalable data synthesis framework. The renderer-based pipeline combines automatically obtained geometric annotations with LLM-generated task instructions and action traces. Its implementation across several modalities demonstrates a potentially useful approach to constructing large-scale interaction datasets with reduced manual annotation effort.
Critical weaknesses
  1. Limited validation of the evaluation metric. CUActSpot evaluates complex actions mainly by checking whether predicted key points fall within predefined regions. However, for dragging and drawing tasks, correct key points do not necessarily guarantee successful execution. The paper lacks validation of how well this metric reflects actual operation success in real GUI environments.
  2. Limited analysis of task and modality diversity. Figure 5 provides evidence that increasing data diversity is more beneficial than simply scaling individual data sources. However, the experiments do not clearly distinguish the effects of task diversity from modality diversity. More controlled comparisons would help clarify the proposed variety-scaling phenomenon.
  3. Lack of quantitative validation of synthetic data quality. The proposed pipeline relies on LLM-generated instructions and action traces, but the paper does not report their error rates or semantic validity. Some synthesis prompts even allow unrealistic interactions, such as dragging non-draggable buttons (Appendix C.1). An analysis of annotation quality and action executability would help establish the reliability of the synthesized data.
Questions for authors

1.Evaluation metric: How well does CUActSpot's region-based metric correlate with actual execution success in real GUI environments, particularly for dragging and drawing tasks? Could the authors provide execution-level validation on a representative subset? 2.Cross-benchmark generalization: Phi-Ground-Any-4B performs well on CUActSpot but relatively poorly on ScreenSpot-Pro and UI-Vision. Although application-specific fine-tuning improves performance on these benchmarks, it also reduces CUActSpot performance. Could the authors provide a more detailed analysis of this trade-off? Is it possible to improve performance across all three benchmarks without sacrificing complex interaction capabilities? 3. Variety scaling: Figure 5 suggests that data diversity is important, but the contributions of task diversity and modality diversity remain unclear. Could the authors provide more controlled experiments to distinguish these effects, ideally under a fixed training budget? 4. Synthetic data quality: Have the authors quantitatively evaluated the accuracy and semantic validity of the synthesized instructions and action traces? In particular, what proportion of the generated actions correspond to executable operations, and how do potentially invalid samples affect downstream performance? Appendix B.2 states that the best checkpoint is selected during training. Could the authors clarify which dataset and metric are used for checkpoint selection? Is a separate validation set used?

1 comment

Thank you for highlighting metric validity and data quality. We address your questions below.

1. Metric versus execution. We replay 80 existing drag/draw cases with our model and GPT-5.4. Region scores agree with application outcomes in 150 of 160 predictions (93.8%). Of 94 predictions accepted by the metric, 88 execute successfully. Of 66 rejected predictions, four execute successfully. For 20 replayable masking cases, mean output-mask IoU is 0.84. We check actual selections, object positions, and output masks, with tool state and mouse-event interpolation fixed. These results support region scoring as a useful execution proxy on this subset. We will include the drag/draw breakdown and replay coverage.

2. Cross-benchmark trade-off. The existing checkpoints show where APP fine-tuning helps and hurts:

CheckpointScreenSpot-ProUI-VisionCUActSpot
Phi-Ground-Any-4B26.315.844.4
+ APP fine-tuning41.529.736.5

Source: paper, Table 2. On CUActSpot, GUI improves by 7.9 points, while text, table, canvas, and image fall by 15.6, 9.4, 7.8, and 14.3 points. The loss is therefore spread across several interaction targets. The click-only APP adaptation favors application grounding at the expense of broader action coverage. A more balanced mixture is a possible remedy, but simultaneous improvement across all three benchmarks remains untested.

3. Task versus modality diversity. Figure 5 changes both diversity and training budget. It supports the benefit of the combined data mixture, but does not separate task diversity from modality diversity. We will revise the variety-scaling claim to reflect this. A fixed-budget comparison is needed to establish their individual effects; we have no additional training runs to report.

4. Synthetic-data quality. Automatic coordinate and trace checks flag 18 of 600 examples (3.0%). Another 42 lack executable semantics or an automatic outcome check. Of the remaining 540, 513 pass replay checks (95.0%). We will report these counts by modality and action. Non-draggable-button examples provide geometric supervision, but should not be described as executable interactions. We will make that distinction explicit. Instruction ambiguity still needs human review, and the downstream effect of removing infeasible examples needs a training ablation. The automatic audit answers the geometry and execution questions, not those two questions.

5. Checkpoint selection. Appendix B.2 needs to state the selection dataset, metric, and whether a separate validation set was used. We will recover these details from the training logs. As a separate check, the final-step checkpoint scores 25.8/15.3/43.8 on ScreenSpot-Pro/UI-Vision/CUActSpot, close to the reported 26.3/15.8/44.4. This evaluates sensitivity to checkpoint choice without selecting another checkpoint by test score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.