Identifiable Protocol States for Cross-Environment GUI Agents
Abstract
GUI agents need to reuse task structure when interfaces change, yet most policies bind progress to screen-specific actions. We introduce Identifiable Protocol States (IPS), an operational abstraction that groups observations by shared preconditions, admissible next subgoals, and recovery options. A proposed match is therefore testable through the behavior it permits. IPS learns this abstraction from successful trajectories, a human-written template of 4–8 subgoals per task family, and automatic language-model slot assignment. A state matcher, invariant miner, protocol-constrained planner, and recovery compiler separate task structure from interface-specific grounding. We also give a conditional sample-complexity argument for reusing a transferred protocol. On SPA-Bench and AndroidWorld, IPS reaches 58.9% and 54.1% task success, compared with 53.7% and 47.4% for GUI-explorer. Controls with the same templates and slot supervision isolate the contribution of cross-environment matching. Matching, recovery, and cost analyses explain where the gains arise and where the abstraction is most useful. The evidence supports protocol-based transfer for structured workflows and shows smaller gains on open-ended browsing.The code is available at https://anonymous.4open.science/r/GUI_Agent-2EE0/.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.