acceptodds
Under review as a conference paper at ICLR 2027

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

Abstract

With the rise of agentic reasoning, visual perception is evolving from passive recognition toward active, task-oriented observation. Active visual perception requires an agent to decide not only what to recognize, but also where to look, how much context to inspect, and when further observation is worthwhile. We explore GUI grounding as a scalable source of spatially grounded supervision for learning such observation behaviors, with the goal of transferring them beyond interface-centric tasks. Our framework first bootstraps observation capabilities from teacher-synthesized agentic GUI Grounding trajectories and then optimizes multi-turn crop-and-inspect decisions with reinforcement learning. Spatial target annotations are transformed into outcome-gated rewards with selective credit assignment, allowing the model to learn not only how to inspect, but also which inspections are useful for task completion. Under fixed inference setting, models based on Qwen3.5-35B-A3B and Qwen3.8-27B obtain consistent improvements over their corresponding baselines. Crucially, after agentic reinforcement learning, these gains transfer to out-of-domain non-GUI benchmarks such as PerceptionBench, HRBench and ChartQAPro, exceeding those obtained from cold-start supervised fine-tuning alone. This result suggests that the learned capability is not merely GUI-specific imitation, but a more general observation strategy. Further analysis reveals a complementary role of the two training stages: SFT acquires the primitives of visual inspection, whereas RL learns to allocate observation effort according to downstream utility. We therefore establish GUI grounding as a scalable supervision paradigm for active visual perception and provide evidence that interface-centric interaction data can induce transferable observation behaviors for broader multimodal reasoning.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.