acceptodds
Under review as a conference paper at ICLR 2027

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

13 views

Abstract

With the rise of agentic reasoning, visual perception is evolving from passive recognition toward active, task-oriented observation. Active visual perception requires an agent to decide not only what to recognize, but also where to look, how much context to inspect, and when further observation is worthwhile. We explore GUI grounding as a scalable source of spatially grounded supervision for learning such observation behaviors, with the goal of transferring them beyond interface-centric tasks. Our framework first bootstraps observation capabilities from teacher-synthesized agentic GUI Grounding trajectories and then optimizes multi-turn crop-and-inspect decisions with reinforcement learning. Spatial target annotations are transformed into outcome-gated rewards with selective credit assignment, allowing the model to learn not only how to inspect, but also which inspections are useful for task completion. Under fixed inference setting, models based on Qwen3.5-35B-A3B and Qwen3.8-27B obtain consistent improvements over their corresponding baselines. Crucially, after agentic reinforcement learning, these gains transfer to out-of-domain non-GUI benchmarks such as PerceptionBench, HRBench and ChartQAPro, exceeding those obtained from cold-start supervised fine-tuning alone. This result suggests that the learned capability is not merely GUI-specific imitation, but a more general observation strategy. Further analysis reveals a complementary role of the two training stages: SFT acquires the primitives of visual inspection, whereas RL learns to allocate observation effort according to downstream utility. We therefore establish GUI grounding as a scalable supervision paradigm for active visual perception and provide evidence that interface-centric interaction data can induce transferable observation behaviors for broader multimodal reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Grounding, Visual Agentic Reasoning, Agentic RL, Think with Image

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EFt5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains crop-and-inspect policies using synthetic GUI grounding tasks. Teacher-trajectory SFT initializes tool use, followed by RL with outcome-gated spatial rewards, a small-target bonus, a textual critic, and selective response updates. The resulting policies improve GUI grounding and transfer to non-GUI perception, high-resolution imagery, and charts.

Critical strengths
  1. Transfer beyond protocol learning. Relative to inspection-capable SFT baselines, RL improves PerceptionBench by approximately 3.0 and 4.5 points across two backbones, with additional gains on HR-Bench and ChartQAPro (Table 2; Figure 2). This supports GUI grounding as a useful supervision source beyond interfaces.
  2. A consequential negative control. Outcome-only RL improves grounding while reducing PerceptionBench performance; the full configuration improves both (Table 3). This usefully demonstrates that source-domain success alone does not guarantee transferable observation behavior.
Critical weaknesses
  1. The distinctive update mechanism is not isolated. Table 3 ablates rewards but does not separately remove spatial masking and surprisal-based filtering. A correctness-plus-critic control without spatial supervision is also missing, limiting attribution to the proposed spatial credit-assignment design.
  2. Improved observation allocation is only indirectly supported. Equal maximum inference budgets do not imply equal actual tool or token use. Appendix F explicitly encourages a successful crop, while the reconstructed histories in Table 4 cannot establish learned stopping behavior. Actual cost and tool-utility measurements are needed for the broader behavioral claims.
Questions for authors
  1. Experimental clarification: What happens when spatial masking and surprisal filtering are removed separately, and when RL uses correctness plus the critic without spatial rewards or masking?
  2. Behavioral clarification: How do SFT and RL compare in actual calls, visual/text tokens, and accuracy at matched realized cost? Do the transfer gains remain stable across repeated training runs?
Reviewer_cQ3YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether GUI grounding can supervise observation behaviors that transfer beyond interfaces. Successful teacher trajectories initialize a crop-and-inspect protocol through SFT. Multi-turn GRPO combines grounding correctness, outcome-gated spatial rewards, a small-target bonus, a textual critic, and selective response masking. Results show consistency improvement on SSPro and PerceptionBench while generalizations on different models and benchmarks.

Critical strengths

1.The supervision source is well motivated: GUI annotations make intermediate observation geometry measurable without specifying ideal crop sequences. 2. Improvements across two backbones, beyond inspection-capable SFT checkpoints, provide meaningful evidence. 3. The paper is clear and easy to read.

Critical weaknesses
  1. The evidence supports the combined recipe more clearly than the masking mechanism. Table 3 does not independently isolate spatial masking or surprisal filtering, and their shared settings across rows are unclear.
  2. Accepted SFT counts of 35k/10k do not establish unique image/task counts, RL dataset size, teacher acceptance rates, or reference-image provenance and deduplication
Questions for authors
  1. From existing manifests and evaluation records, can the authors specify the training/evaluation populations and safeguards separating training and checkpoint selection from testing?
  2. Which masks are enabled in each Table 3 configuration, and what attribution to masking is supported by those existing comparisons, as distinct from the full reward recipe?
Reviewer_K9EUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses GUI grounding as supervision for active visual observation. A teacher first generates successful multi-turn crop-and-inspect trajectories; supervised fine-tuning teaches the interaction protocol, and GRPO then optimizes a reward combining final grounding correctness, spatial process feedback, a small-target bonus, format checks, and a text-based trajectory critic. On Qwen3.5-35B-A3B and Qwen3.8-27B, RL improves ScreenSpot-Pro over the SFT checkpoints by 1.83 and 3.99 percentage points. The trained observation policy also improves PerceptionBench and transfers to OSWorld-G-R, HR-Bench 4K/8K, and ChartQAPro.

Critical strengths
  1. I think using GUI grounding data to train general visual observation is interesting. The transfer beyond GUI tasks is the strongest part of the paper.
  2. The evaluation covers two model backbones and several non-GUI benchmarks. The results on PerceptionBench, HR-Bench, and ChartQAPro provide some evidence of cross-domain transfer.
  3. The reward ablations are useful. They show that outcome-only RL mainly helps GUI grounding, while intermediate observation rewards help transfer.
  4. The observation-feedback experiments are also helpful. They show that high-resolution crops matter more than crop metadata alone.
Critical weaknesses
  1. The method is not very novel. DeepEyes already trains VLMs to use crop-and-zoom tools with RL. RegionFocus and GUI-Eyes already study active visual inspection for GUI grounding.

  2. The gains over SFT are modest. RL improves ScreenSpot-Pro by only 1.83 and 3.99 points, and PerceptionBench by 3.0 and 4.5 points on the two backbones. Also the paper does not report confidence intervals making the gains unclear. Also comparisons with prior active-perception methods do not control for the backbone / training data / inference budget. This makes it difficult to isolate the contribution of the proposed method.

Questions for authors
  1. What is the main technical difference between the proposed method and prior crop-and-zoom methods such as DeepEyes, RegionFocus, and GUI-Eyes?
  2. Could the authors compare with these methods using the same backbone, training data, and inference budget?
  3. Could the authors report results across multiple training runs? This is especially important because the gains over SFT are relatively small.
  4. Why does the full reward only achieve 80.39 on ScreenSpot-Pro, while “ungated process + critic” achieves 81.09?
2 comments

Thank you for your thoughtful review and constructive feedback. We would be grateful to hear whether our rebuttal has addressed your concerns regarding the technical contributions, experimental comparisons, and reliability of the improvements.

As the discussion period approaches its conclusion, we would greatly appreciate any follow-up feedback. If any concerns remain, please let us know so that we can address them. Thank you again for your time and careful consideration of our work.

Dear Reviewer K9EU,

Thank you for your thoughtful feedback over the past few days. We have carefully considered your comments and addressed the points you raised through additional experiments, analyses, and manuscript revisions.

As the rebuttal period draws to a close, we hope our responses have adequately addressed your concerns. We will incorporate the discussed clarifications and additional results into the final manuscript.

Thank you again for your time and valuable feedback again.

Sincerely,

The Authors of Submission 32691

Comments on the paper

1 comment

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely thank you for your valuable and thoughtful comments. We appreciate the reviewers’ recognition of GUI grounding as a promising supervision source for visual observation, the transfer improvements beyond GUI tasks, and the informative reward and visual-feedback analyses.

The main concerns involve the distinction from prior crop-and-inspect methods, the strength of the experimental comparisons, the contribution of selective response updates, and the evidence for observation efficiency. In response, we have clarified our observation protocol and supervision design, added same-backbone comparisons with adapted RegionFocus, DeepEyes, and GUI-Eyes, and conducted an additional complete SFT-and-RL training run, which shows consistent improvements in both GUI grounding and non-GUI perception.

We have also provided joint filtering ablations and a correctness-plus-critic control to better characterize the contributions of the training components. Additional measurements report actual model calls and generated completion lengths, complementing the existing analysis of the usefulness of high-resolution crop images. We further clarified training-data provenance, shared experimental settings, and checkpoint and reward-configuration selection procedures.

Detailed explanations and supporting results are available in our point-by-point responses. We will consolidate these additions in the revised manuscript and publicly release the final model checkpoint and complete evaluation protocols to support reproducibility.

We would greatly appreciate the reviewers’ follow-up feedback on whether our responses have addressed their concerns, so that we can clarify any remaining questions before the discussion period ends.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.