- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CUActSpot, a manually constructed benchmark for evaluating complex computer-use action grounding across five modalities: GUI, text, table, canvas, and natural image. It contains 206 samples spanning clicking, dragging, and multi-point drawing, with evaluation based on correct regions, banned regions, and ordering constraints. Separately, the paper proposes a data-synthesis pipeline that provides an LLM with screenshots and structured spatial annotations to generate instructions and corresponding action traces. The authors construct approximately 50 million synthetic samples and train Phi-Ground-Any-4B using a mixture of this data and OpenCUA data. Evaluation includes CUActSpot, ScreenSpot-Pro, UI-Vision, and OSWorld, together with ablations examining the effects of training-data diversity.
Critical strengths
- Considering a broader range of interaction targets and actions is interesting and practically relevant. The focus on dragging, drawing, text selection, and table manipulation goes beyond locating standard GUI buttons. The initial failure analysis provides a concrete motivation for studying these interactions rather than continuing to focus primarily on single-click grounding.
- The correct-region and banned-region formulation looks reasonable. Supporting both ordered and unordered key points is a useful extension for evaluating more complex interactions. Banned regions also address a potential weakness of multi-point evaluation, where predicting excessive coordinates could otherwise inflate success rates.
Critical weaknesses
- The technical novelty appears limited, making the quality and validation of the data contributions particularly important. I am not yet convinced that the model or training procedure constitutes a substantial technical advance beyond extending the training distribution to additional modalities and actions.
- The reliability of the synthetic instructions and action traces is insufficiently validated. The pipeline does not ask the generator to recover all coordinates from screenshots alone: it supplies structured spatial annotations and allows the model to calculate additional coordinates. Correct element coordinates do not guarantee that the generated instruction is unambiguous, that the coordinate arithmetic is correct, or that the action would produce the intended result in an actual application.
Questions for authors
- Can you provide a more detailed statistical comparison with existing benchmarks? Table 1 groups prior benchmark into one row. I would like to see separate detailed comparisons covering sample counts, applications, action-type distributions, and interaction targets under a consistent taxonomy. For CUActSpot, please also report the number of examples in each modality–action category and explain how these categories contribute to the overall score.
- How much of the improvement comes from the proposed synthesis pipeline under a controlled training setup? Please compare existing data, the proposed synthetic data, and their combinations using the same backbone, training-token budget, visual-input settings, and OpenCUA mixture. Report results across CUActSpot and the external benchmarks.
- What are the synthetic-data error rates, and how are invalid examples handled? Please report human-checked instruction and trajectory correctness by modality and action type. An ablation removing deliberately infeasible interactions would help determine whether these examples provide useful geometric supervision or introduce misleading action semantics.
- How well does the automatic score correspond to actual interaction success? Can you compare the region-based metric with human judgments or replayed execution, particularly for multi-point drawing and masking?
Thank you for the detailed suggestions. The main contributions are the benchmark and synthesis pipeline. Phi-Ground-Any is a reference model trained with a standard recipe; we will make this clearer.
1. Benchmark statistics. We will replace the grouped row in Table 1 with separate benchmark entries:
Coverage is listed as applications or domains according to the source; five domains does not mean five applications. The two ScreenDrag benchmarks will be listed separately. We will use the same action and target categories for each benchmark and add action-type counts. For CUActSpot, we will report all 15 modality–action cells, including empty cells, and state how their sample counts enter the overall score. These statistics will come directly from the annotations and scoring code.
2. Contribution of synthetic data. Appendix Table 8 reports the following existing results:
In the target additional evaluation, the GUI-only and full-mixture checkpoints score 10.0% and 14.5% on UI-Vision. The broader mixture helps across all three benchmarks in this scenario. However, sample budget also grows, so this comparison does not isolate the effect of synthesis or diversity. We will state this clearly and list the backbone, token budget, visual settings, and OpenCUA share for each existing run. The requested matched-budget comparison of existing data, synthetic data, and their combination requires new training and remains open.
3. Instruction and trajectory quality. Structured coordinates make synthesis easier, but they do not ensure a valid instruction or trajectory. In the target audit of 600 examples, coordinate and trace checks flag 18 (3.0%). Another 42 lack executable semantics or an automatic outcome check. Among the remaining 540, 513 pass replay checks (95.0%). We will break these results down by modality and action, and describe how flagged examples are handled. These checks cover arithmetic, trace structure, and verifiable outcomes. They do not measure instruction ambiguity, which still needs human checking. We will also label deliberately infeasible interactions as geometric supervision. Their effect on the trained model remains untested because we have no removal-and-retraining ablation.
4. Region scores versus actual success. In the target replay study, our model and GPT-5.4 act on 80 existing drag/draw cases. Region scores agree with application outcomes for 150 of 160 predictions (93.8%). Six of 94 accepted predictions fail execution; four of 66 rejected predictions succeed. On 20 replayable masking cases, mean output-mask IoU is 0.84. This checks the resulting selection or mask, beyond whether key points enter the correct regions. We will report drag and draw separately and describe the replay coverage, tool states, and interpolation settings.
Sources: ScreenSpot-Pro, UI-Vision, Beyond Clicking, ShowUI-π.