acceptodds
Under review as a conference paper at ICLR 2027

Identifiable Protocol States for Cross-Environment GUI Agents

Abstract

GUI agents need to reuse task structure when interfaces change, yet most policies bind progress to screen-specific actions. We introduce Identifiable Protocol States (IPS), an operational abstraction that groups observations by shared preconditions, admissible next subgoals, and recovery options. A proposed match is therefore testable through the behavior it permits. IPS learns this abstraction from successful trajectories, a human-written template of 4–8 subgoals per task family, and automatic language-model slot assignment. A state matcher, invariant miner, protocol-constrained planner, and recovery compiler separate task structure from interface-specific grounding. We also give a conditional sample-complexity argument for reusing a transferred protocol. On SPA-Bench and AndroidWorld, IPS reaches 58.9% and 54.1% task success, compared with 53.7% and 47.4% for GUI-explorer. Controls with the same templates and slot supervision isolate the contribution of cross-environment matching. Matching, recovery, and cost analyses explain where the gains arise and where the abstraction is most useful. The evidence supports protocol-based transfer for structured workflows and shows smaller gains on open-ended browsing.The code is available at https://anonymous.4open.science/r/GUI_Agent-2EE0/.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.