acceptodds
Withdrawn from ICLR 2027

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Constructing robust GUI agents relies on multimodal visual grounding—the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model’s grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, GUI agents, multimodal large language models, visual grounding, referring expression comprehension, reinforcement learning, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sqDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScreenRef, a large-scale GUI grounding dataset with 586K screenshots and 633K referring expressions across desktop, web, and mobile platforms. Based on this dataset, the authors propose D2RL, which applies on-policy self-distillation before reinforcement learning, and train GroundGUI-8B. The model achieves competitive performance on several GUI grounding benchmarks.

Critical strengths
  1. The data construction pipeline is complete and carefully designed.
  2. The experimental evaluation is comprehensive. The paper evaluates GroundGUI-8B on five GUI grounding benchmarks and includes both dataset-level comparisons and ablation studies for the proposed training pipeline.
Critical weaknesses
  1. Insufficient motivation for ScreenRef and D2RL. The paper claims that existing datasets lack diversity or annotation quality and that RL suffers from sparse rewards, but these issues are not sufficiently quantified before introducing the new dataset and training method. As a result, the necessity of both contributions is not fully convincing.

  2. Relatively modest gain from D2RL. From the ablation study, the additional benefit of combining the two stages appears limited.

Questions for authors
  1. Can the authors provide more direct evidence that existing GUI grounding datasets are limited in diversity or annotation quality?

  2. How severe is the sparse-reward problem in practice?

  3. Given the relatively small gain over OPSD or RLOO alone, what is the main advantage of the D2RL method?

Reviewer_5ukyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GroundGUI studies grounding language instructions to interface elements across desktop, web, and mobile environments, including abstention when a requested target is absent. Its motivation is limited access to diverse, reliable public training data and the difficulty of moving from supervised imitation to sparse-reward policy learning. The paper constructs ScreenRef through screenshot-metadata processing and tutorial-video analysis, followed by automatic instruction generation, coordinate annotation, and verification. A vision-language model is first supervised on this resource. It then undergoes on-policy self-distillation: a frozen teacher receives a highlighted target and a textual localization hint, while the student predicts from the original screenshot and instruction. Token-distribution matching emphasizes spatially significant coordinate digits. Subsequent leave-one-out policy optimization uses grounding success on the original screenshot. At inference, the trained model produces a grounding response without privileged annotations or the teacher.

Critical strengths
  1. Broad public GUI data and absent-target supervision are relevant needs, and the paper explains why both data coverage and a better RL starting policy matter.

  2. Evaluation uses established GUI benchmarks and includes dataset comparisons, isolated post-training stages, and an extra-SFT control. A validated resource would have independent community value.

Critical weaknesses
  1. The motivation for public GUI data is clear, but the central high-quality-resource contribution is not independently established. Annotation and verification rely on automated models and confidence thresholds without a representative human accuracy/ambiguity audit or calibration of the selection confidence. Source-level filtering, overlap checks, and error distributions remain insufficiently documented. Dataset comparisons use separately selected settings rather than matched data/compute budgets, so improved downstream scores alone cannot establish superior annotation quality. This evidential gap directly affects the principal resource claim.

  2. The methodological novelty of D2RL is limited. Visually privileged self-distillation and coordinate-significance-aware supervision already appear in GUI-SD; the main increment is staging such distillation before established leave-one-out RL. The gain over the extra-SFT/RLOO control is only 0.8 points on the reported average, without uncertainty or comparable teacher/compute accounting. The weighting strength, tokenizer-level digit treatment, post-training subset composition, and reward handling for refusal cases are not fully specified in the manuscript. The current evidence therefore supports a useful recipe more strongly than a substantial new learning methodology.

Questions for authors
  1. Please provide a source/platform/task-stratified human annotation audit, confidence calibration, filtering statistics, and overlap checks supporting the quality claim.

  2. What are the coordinate-weighting strength and tokenizer mapping, post-training subset composition, and teacher targets/rewards for refusal examples?

  3. Does D2RL outperform the extra-SFT/RLOO control across runs at comparable total cost? Why use only the chosen backbone, and can available open-source baseline gaps be filled?

  4. What privacy audit, redaction, and source-permission checks will determine the releasable data subset?

Reviewer_ULicOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GroundGUI, an approach to improving GUI grounding through better training data and multi-stage training. Specifically, it introduces ScreenRef, a dataset built from structured UI information and tutorial videos across desktop, web, and mobile interfaces. The model is trained with SFT, followed by OPSD with privileged target information (i.e., self distillation from the SFT model), and finally RL. The resulting model shows improved performance across a range of GUI grounding benchmarks.

Critical strengths

None

Critical weaknesses
  1. Training method is not new. Training with SFT -> OPSD -> RL is well known technique in the field, e.g., Kimi 3.

  2. Unclear novelty and extensibility of the data construction pipeline. Given the limited novelty of the training recipe, the main contribution is the dataset. However, I view ScreenRef as a one-time dataset construction effort rather than a scalable data-generation methodology. Once a model has been trained on this dataset, it is unclear how the proposed approach continues to provide useful supervision. A more meaningful contribution would be a data transformation pipeline that identifies examples current models struggle with and continually evolves as their capabilities improve. Instead, the paper combines existing collection and annotation pipelines to produce a fixed dataset. While that dataset may be useful, I do not see a sufficiently novel methodology for sustaining improvements beyond this initial round of training.

  3. Unverified downstream value for GUI/web agents. The main goal of GUI model is to improve web agent (not GUI prediction). For the rebuttal, I kindly request the authors to add realistic web agent experiment.

Questions for authors

See the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.