acceptodds
Under review as a conference paper at ICLR 2027

VISTA: View-Consistent Self-Verified Training for GUI Grounding

Abstract

When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rolloutsvare sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA (View-Consistent Self-Verified Training), a GRPO-based training framework that constructs each comparison group from multiple target-preserving views of the same GUI instance. Each view is generated by a crop that keeps the target element visible and remaps its box exactly, so model rollouts are compared across semantically equivalent but geometrically different inputs. To stabilize short coordinate generation without turning reinforcement learning into unconditional imitation, VISTA further adds a self-verified crossview anchor: an oracle answer optimized with an advantage-weighted loss, excluded from the group baseline and activated only when the model has produced a maximum-reward rollout. Across five GUI-grounding benchmarks and multiple Qwen backbones, VISTA consistently improves grounding accuracy. On ScreenSpotPro, it raises Qwen3-VL 4B/8B/30B-A3B from 55.5/52.7/53.7 to 63.4/65.8/67.0. Robustness analyses further show higher worst-view accuracy and lower prediction flip rates.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Grounding, RL, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YbnQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VISTA studies GUI grounding from a screenshot and a language instruction, with a click coordinate as output. It addresses the loss of useful relative advantages when fixed-view reinforcement-learning rollouts are uniformly successful or unsuccessful. During training, target-preserving crops create different views of the same annotated instance, and the target box is remapped into each view. Model-generated coordinate responses across these views form a shared comparison group with point-in-box rewards. The method adds a ground-truth center-coordinate anchor whose weight is positive only when the group contains both successful and unsuccessful model responses. The anchor is excluded from model-only group statistics and is optimized through an advantage-weighted supervised term alongside the policy objective. At inference, the trained model predicts a click directly from the screenshot and instruction; an optional inference-time view-aggregation procedure can be used separately.

Critical strengths
  1. The introduction identifies a concrete optimization bottleneck, and the method separately specifies cross-view group construction and selective anchoring without changing the model-only baseline.

  2. Experiments cover established GUI benchmarks, multiple model scales, and meaningful crop-augmentation and component controls. These comparisons support a useful training contribution.

Critical weaknesses
  1. The technical increment over augmentation and existing oracle-guided policy optimization is narrower than the broad framework presentation suggests. LUFFY and BREAD already combine policy learning with external guidance, although their mechanisms differ from VISTA. The grounding-specific contribution is the cross-view grouping and gated anchor combination. Grouping alone adds only 0.3 points over the crop-augmentation control, while the full method incurs additional training time. Missing repeated-run uncertainty and matched-compute comparisons make the incremental value difficult to quantify. Failure of an ungated anchor with an extreme normalization weight is also not a sufficient test of the gating principle against a numerically stable alternative.

  2. The construction guarantees target visibility and coordinate consistency, but not preservation of instruction meaning: a crop can remove a relational anchor or alter a relative-order description. This assumption is central to treating the views as equivalent learning instances and is not directly audited. The robustness diagnostic uses training examples and the training crop distribution, as acknowledged in the paper, so it cannot establish general robustness to unseen view changes.

Questions for authors
  1. Please compare the full method and augmentation control at matched compute, with repeated-run uncertainty and a stably weighted ungated anchor.

  2. How often do crops invalidate referring instructions? Please provide a focused validity audit and a held-out view diagnostic.

  3. How were the data mixture and checkpoint selected, and why were the chosen backbone families used? Please fill available open-source baseline results where reproducible.

Reviewer_wCMQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VISTA modifies GRPO for GUI grounding by grouping one rollout from each of eight target-preserving screenshot crops. It also adds a ground-truth box-center anchor, trained only when at least one rollout succeeds and excluded from group statistics. The paper reports fewer degenerate groups, improved grounding accuracy across Qwen3-VL scales and Qwen3.5 backbones, and greater crop consistency.

Critical strengths
  1. The paper identifies a concrete training problem: binary point-in-box rewards leave many GRPO groups without a useful relative signal. The multi-view grouping and gated anchor form a simple recipe, and the anchor ablation helps explain why gating matters.

  2. Evaluation across three model scales, a second backbone family and five benchmarks gives the practical findings useful breadth. The reported additional wall-clock cost is moderate, making the method potentially easy to adopt.

Critical weaknesses
  1. The incremental gain over matched GRPO baselines is modest and its reliability is unclear. Standard GRPO accounts for much of the improvement over the base models. On the 8B model, GRPO with crop augmentation reaches 64.0 versus VISTA's 65.8 on ScreenSpot-Pro, while view grouping alone contributes only 0.3 points. With one training seed and no uncertainty estimates, these results do not yet establish a reliable advantage for each component. The relevant evidence is paired evaluation and variation across training seeds, rather than base-model improvement alone.

  2. The distinct contribution of cross-view grouping and the oracle anchor needs clearer isolation. With one rollout per view, groups mix different input difficulties instead of comparing multiple responses to the same input. The consequences for the estimator should be explained, and compared with multiple rollouts per view at fixed total budget. The anchor also uses ground-truth supervision, so its benefit should be separated from a well-tuned mixed RL/SFT baseline. Finally, training-example consistency under the training crop distribution is limited evidence for broader robustness.

Questions for authors
  1. Are standard GRPO and crop-augmented GRPO matched in data, steps and hyperparameters? Please provide paired uncertainty estimates and, if available, results across training seeds for the modest incremental gains.

  2. At a fixed eight-rollout budget, how does K=4, n=2 compare with K=8, n=1? Please explain how view difficulty affects the group-relative advantages and what objective the resulting update estimates.

  3. Can the gated anchor be compared with a tuned mixed RL/SFT control using the same ground-truth targets and compute? Please distinguish training-crop consistency from held-out transformation robustness and single-view inference from +MVP results.

Reviewer_nabEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents VISTA, a reinforcement learning framework for mitigating reward degeneracy in Group Relative Policy Optimization (GRPO) for GUI grounding. The authors argue that sampling repeated rollouts from a single screenshot can produce all-fail or all-success groups, leaving little or no relative learning signal, particularly for coordinate-sensitive UI targets. VISTA instead constructs groups from multiple target-preserving crops of the same GUI instance, with the target box remapped into each crop’s coordinate frame. It also adds a self-verified cross-view anchor: an oracle center-point target is applied to an unsuccessful view only when the model-generated group contains both successful and unsuccessful rollouts; the anchor is excluded from the GRPO group statistics. Experiments on five GUI-grounding benchmarks with Qwen3-VL and Qwen3.5 backbones show overall improvements, with particularly notable gains on ScreenSpot-Pro.

Critical strengths

Accurately identifies and targets a fundamental bottleneck (reward degeneracy/variance collapse) in applying GRPO to fine-grained, coordinate-sensitive GUI grounding tasks.

The proposed view-consistent group rollout is an elegant, zero-additional-human-annotation solution that transforms single-view sampling into diverse multi-view geometric perturbation.

The self-verified cross-view anchor design is conceptually sound, effectively balancing autonomous exploration in reinforcement learning with stable coordinate convergence via conditional gating.

Extensive empirical validation across diverse model scales (4B to 35B MoE) and difficult benchmarks (especially ScreenSpot-Pro) demonstrates the effectiveness of the framework.

Critical weaknesses

Relying on target-preserving crops creates a structural mismatch between training-time localized views and full-screen UIs during real-world inference. While cropping helps optimization, it does not fundamentally resolve full-screenshot reasoning, where an end-to-end dynamic zoom-in mechanism (e.g., GPT-o3-style visual search) would be more principled.

Localized cropping carries the risk of stripping away critical global contextual cues necessary for disambiguating intent in dense UI environments.

The paper reports an approximately 25% increase in wall-clock training time over GRPO, but provides limited detail on absolute compute cost, such as GPU hours, throughput, and memory usage, making the practical scalability of VISTA difficult to assess.

Qualitative analyses regarding remaining failure modes (e.g., cases where multi-view rollouts fail to rescue all-fail states) are currently missing.

Questions for authors

Have the authors conducted an analysis on how context-sensitive tasks are affected by localized cropping? How does the model perform when critical global cues (like panel headers or navigation bars) are cropped out?

The paper fixes the initial crop size at (0.9W \times 0.9H). How sensitive is VISTA to the crop size and the distribution of crop positions, beyond the reported ablations on (p_{\text{crop}}) and the number of views (K)?

Could the authors report how the gate activation rate ((g=1)) and the rate of groups receiving a positive anchor advantage (mixed-reward groups) change over training, including across different data or task categories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.