acceptodds
Under review as a conference paper at ICLR 2027

Learn From Move: Can GUI Agents Learn How Their Environments Work?

Abstract

GUI benchmarks increasingly emphasize longer workflows, but task horizon alone does not capture whether agents can learn how an environment responds to their actions. We introduce LatentGUIWorld, a benchmark of six tasks in three matched pairs organized by the evidence needed for control: uses prior knowledge and available observations, requires interaction that directly reveals hidden semantics, and identifies dynamics from action–response relationships. Frontier models struggle especially on tasks. We introduce MOVE, which separates movement from button events and exposes intermediate visual feedback, together with demonstrations of iterative exploration and control. Ablations show that both action granularity and multi-step demonstrations improve performance. After SFT, LatentLearner-9B achieves 50.0% overall success on LatentGUIWorld, comparable to several frontier models. Joint RL further raises egocentric Drag and Ten-Choice success to 92.0% and 75.3%. Behavioral analysis reveals multi-step exploration. In egocentric Drag, RL increases Pearson correlation between error-normalized commands and ideal gain compensation from 0.610 to 0.803. Together, these behaviors show that models learn to explore through interaction and adapt their control to environment dynamics. The same format improves GUI grounding after RL, reaching 67.37% mean accuracy across five benchmarks versus 66.72% for conventional RL.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agents, Learning from Interaction, Environment Identification, Adaptive Control, GUI Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UNwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LatentGUIWorld. It consists of 6 synthetic GUI environments organized around 3 evidence levels, namely available context, interaction that reveals hidden semantics, and action–response evidence for identifying dynamics. It proposes MOVE, which separates cursor movement from button events and provides observations between primitives. A 9B model trained on scripted demonstrations achieves 50.0% overall success. Additional experiments examine gain compensation and standard GUI grounding.

Critical strengths
  1. Short tasks with hidden response properties can expose weaknesses that aggregate success on long workflows may obscure. In this sense, the evaluation question is valuable.
  2. Egocentric Drag improves from 41.3% after SFT to 92.0% after paired RL, which is significant.
Critical weaknesses
  1. The benchmark currently classifies tasks by the minimum evidence required for reliable success, but the constructions mainly establish that different hidden gains require different ideal one-step corrections. For instance, egocentric Drag uses a positive isotropic gain of g ∈[0.5,0.8]∪[1.2,1.5]\in [0.5, 0.8] \cup [1.2,1.5] and under the stated linear law, consider a controller that simply commands the current visible residual, ut=etu_t=e_t, without estimating gg, an unclamped movement will have et+1=et–gut=(1−g)ete_{t+1}=e_t – gu_t = (1-g)e_t where ∣∣et+1∣∣≤0.5∣∣et∣∣||e_{t+1}|| \le 0.5 ||e_t||. Thus, repeated visual correction can converge across the entire evaluated gain range without identifying the episode-specific gain or consulting previous action–response pairs. It does not establish that successful control requires dynamics identification.
  2. The ablations change task capability and supervision, thus weakens their interpretation. For instance, replacing MOVE with atomic click/drag simultaneously changes action granularity, intermediate observations, and opportunities to inspect feedback before commitment. Removing standalone movement may therefore remove a useful information-acquisition operation, rather than isolate a learning advantage.
  3. Some of the related methodological prior works are absent from the experiments, including GUI-Cursor [1] which already trains iterative cursor movement using rendered visual feedback and multi-step RL.
Questions for authors
  1. Does performance deteriorate when previous action magnitudes are removed or corrupted while screenshots remain available?
Reviewer_7kWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LATENTGUIWORLD, with six GUI tasks in three pairs, classified by the evidence needed for control: I0 (prior knowledge and current observations suffice), I1 (interaction directly reveals hidden semantics), and I2 (environment dynamics must be inferred from action–outcome pairs). It proposes MOVE, which separates cursor movement from button events and returns visual feedback after actions, and trains LatentLearner-9B using multi-step exploration/control demonstrations. The abstract reports 50.0% overall success after SFT; joint RL raises egocentric Drag and Ten-Choice success to 92.0% and 75.3%; the Pearson correlation between Drag command magnitude and ideal gain compensation rises from 0.610 to 0.803; and mean accuracy across five grounding benchmarks is 67.37% with RL-formatted actions versus 66.72% with conventional RL.

Critical strengths

1.The problem is concrete and the paired benchmark is diagnostic. Separating task length from the evidence required for correct control, and instantiating the distinction with three task families, can help diagnose GUI-agent capability gaps.

2.The method and reported validation are aligned. MOVE exposes intermediate feedback, and the abstract reports action-granularity/multi-step-demonstration ablations, joint RL, and grounding transfer rather than a single benchmark score alone.

Critical weaknesses
  1. The abstract does not clearly distinguish in-episode control identification from prior environment-transition learning. Clarify that the paper studies identifying episode-specific continuous gain or rotation mappings from action–outcome feedback, and explain how its formulation and evaluation differ from prior work.

  2. Evidence for learning the dynamics is indirect. The abstract reports success rates and correlations, but no hidden-parameter recovery or generalization to unseen dynamics. The 0.65-point grounding gain also lacks statistical uncertainty.

  3. The novelty boundary needs clearer positioning against prior work. GTTA explores environments and extracts transition rules; GUI-Cursor uses post-MOVE feedback for iterative grounding; GUI-explorer mines transition knowledge; UI-Oceanus and STP train on GUI state transitions. The contribution appears narrower: paired tasks for identifying episode-specific continuous control mappings. Acknowledge these precedents and avoid implying that MOVE feedback or GUI dynamics learning is new. GTTA, GUI-Cursor, GUI-explorer, UI-Oceanus, and STP.

Questions for authors
  1. Please compare the identification targets and feedback use in GTTA, GUI-Cursor, and this work. Do GTTA’s discrete transition rules cover this paper’s episode-specific continuous gain/rotation parameters? What identification ability does feedback during a pressed drag add beyond GUI-Cursor’s cursor-localization feedback?
  2. Do you directly evaluate recovery error for hidden gain/rotation mappings? Does the model generalize to unseen gain ranges or control mappings?
  3. Is the difference between 67.37% and 66.72% statistically significant across multiple seeds or benchmarks? What are the per-benchmark results and variances?
Reviewer_TqtsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LatentGUIWorld, six synthetic GUI tasks organized into matched Drag, Ten-Choice, and Rotation variants. Its taxonomy distinguishes information available from context, semantics revealed by interaction, and dynamics inferred from action–response relationships. MOVE separates cursor movement from button events and supplies intermediate screenshots. Scripted demonstrations and subsequent task-pair RL train feedback-conditioned control. An additional grounding experiment finds a modest advantage after RL.

Critical strengths
  • The benchmark asks a useful question that task horizon alone misses: whether agents exploit evidence created by their own actions. Explicit control laws, paired scenes, strict Rotation scoring, disjoint validation-based checkpoint selection, and documented content holdouts make the experimental setting comparatively interpretable. The exploration-prompt experiment also usefully separates unguided performance from performance with task-specific guidance.

  • The empirical improvements are substantial on the egocentric tasks. The paper goes beyond terminal success with an error-normalized gain-compensation analysis, includes unsuccessful movement episodes, checks filtered cohorts, and compares shared episode IDs. These qualifications make the behavioral evidence more informative than isolated successful trajectories. Paired RL retains strong exocentric performance while improving egocentric control.

Critical weaknesses
  • The evidence supports effective feedback control more directly than identification of an environment model. Drag and Ten-Choice gains remain within the training ranges; their visual/content holdouts do not test new dynamics. Command–inverse-gain correlations, even with the reported robustness checks, do not distinguish explicit or implicit gain estimation from reactive correction policies operating within familiar gain bands. The taxonomy defines levels through the minimum evidence sufficient for reliable success, yet no simple controller comparison establishes that latent identification is necessary under the actual budget. This matters because environment identification is the central scientific framing.

  • The ablations do not fully isolate the proposed learning mechanism. Removing MOVE also removes the natural hover operation required to reveal Ten-Choice messages, so that task's collapse partly reflects lost interaction capability. The one-move demonstrations change exploration coverage as well as trajectory structure. Similarly, grounding gives MOVE multiple predictions and cursor observations while the direct-click policy predicts once; its 0.65-point mean improvement has no reported uncertainty or equal-inference-budget comparison. These results demonstrate useful configurations, but provide weaker evidence for the specific contribution of action–response learning.

Questions for authors
  1. Can a simple visual feedback controller without episode-level gain estimation solve the egocentric tasks within 12 turns? How does the learned policy perform on held-out gain ranges or response signs?
  2. Can you retain hover/revelation capability while removing useful action–response history, and compare grounding against an equally budgeted iterative direct-click predictor? Paired confidence intervals would help interpret the smaller gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.