acceptodds
Withdrawn from ICLR 2027

Identifiable Protocol States for Cross-Environment GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI agents need to reuse task structure when interfaces change, yet most policies bind progress to screen-specific actions. We introduce Identifiable Protocol States (IPS), an operational abstraction that groups observations by shared preconditions, admissible next subgoals, and recovery options. A proposed match is therefore testable through the behavior it permits. IPS learns this abstraction from successful trajectories, a human-written template of 4–8 subgoals per task family, and automatic language-model slot assignment. A state matcher, invariant miner, protocol-constrained planner, and recovery compiler separate task structure from interface-specific grounding. We also give a conditional sample-complexity argument for reusing a transferred protocol. On SPA-Bench and AndroidWorld, IPS reaches 58.9% and 54.1% task success, compared with 53.7% and 47.4% for GUI-explorer. Controls with the same templates and slot supervision isolate the contribution of cross-environment matching. Matching, recovery, and cost analyses explain where the gains arise and where the abstraction is most useful. The evidence supports protocol-based transfer for structured workflows and shows smaller gains on open-ended browsing.The code is available at https://anonymous.4open.science/r/GUI_Agent-2EE0/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Identifiable Protocol States (IPS), GUI agents, Protocol discovery, Cross-environment transfer, Task abstraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wtPXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Identifiable Protocol State in whcih GUI observations are grouped according to shared preconditions, admissible next subgoals, and recovery options. The system combines a contrastive-trained state matcher, invariant checks, graph-based planning, and recovery. Training uses successful trajectories, human-written family templates, and LLM-generated slot labels.

This submission has gone beyond the page limit. The main text should be 9 pages or fewer, but this submission has 10.

Critical strengths
  1. The paper distinguishes operational agreement from formal statistical identifiability. It treats its learning bound as conditional and acknowledges that matching-success correlations are descriptive. This is an honest acknowledgement of limits.
  2. Distinguishing task stage from screen appearance is a sensible approach to GUI transfer.
Critical weaknesses
  1. Section 3.2 trains the matcher using same-slot assignments, while execution requires family-specific objects, including states, centroids, invariant sets, and graph protocal. Section 4.1 says that it excludes unseen-family trajectory from matcher and invariant-miner training. But the introduction states that a new family requires its own template and suitable trajectories.
  2. The reported contrastive objective supervises the fused embedding using template-slot labels only. I don't think it separately constrain the heads to encode preconditions, next subgoals, and recovery structure like what the authors describe in Eq. 4. They do not necessarily learn those concepts. Table 3 supports the combined representation, but does not validate the decomposition.
  3. The theoretical analysis assumes the frozen protocol already permits low target risk and that the restricted class has the desired capacity, which may be less practical. Also, the bound estabilshed doesn't connect signature agreement to transferability, bound the effect of matching error, or demonstrate an empirical sample-efficiency improvement.
  4. It has gone beyond the page limit for main text.
Questions for authors

See weakness

Reviewer_N8jYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GUI agents often break when interfaces change because their policies tie progress to screen-specific actions. This paper proposes Identifiable Protocol States (IPS), which defines a task stage by three operational signatures: preconditions, admissible next subgoals, and recovery options. Observations that share these signatures belong to the same state, so a cross-interface match can be tested through behavior rather than visual similarity. Effectiveness was evaluated in terms of task success, operational validity, robustness, and execution costs.

Critical strengths
  • A clear, falsifiable definition of task-stage equivalence based on preconditions, next subgoals, and recovery options rather than visual similarity.
  • Well-designed controls to isolate procedural supervision from cross-environment matching.
  • The learned representation is evaluated directly, through operational agreement, a human audit, and the effect of matching errors on execution.
  • Protocol-level recovery clearly improves on action-level retry, and the gains hold with an open-source backbone.
Critical weaknesses
  • Heavy setup with limited transfer: each task family needs a human template. Transfer is shown mainly within known task families.
  • The state set is closed, with no handling of unmatched screens. Mismatches substantially hurt execution, and only 58.4% of them are caught.
  • No comparison with lighter alternatives, such as skill libraries or experience retrieval annotated with preconditions and recovery, so the need for the added complexity is unclear.
Questions for authors

Q1: How is a protocol state assigned at inference time (for example, nearest centroid)? What happens when the current screen does not correspond to any learned state? Did you consider a confidence threshold, an "unknown state" option, or a fallback to the base action model, and how often do out-of-distribution screens occur in the unseen-family setting?

Q2: How does IPS compare with a skill-library or experience-retrieval baseline in which each reusable unit is annotated with preconditions, postconditions, and recovery options, mined automatically from the same trajectories without per-family templates, and checked at run time by the action model? Which of IPS's gains depend specifically on the trained contrastive matcher?

Q3: Can the approach transfer to desktop environments such as OSWorld or Mind2Web? What is the expected cost of scaling to many task families, including routing a new task to the right family?

Reviewer_UJsGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed Identifiable Protocol States (IPS), an operational abstraction that groups observations by shared preconditions, admissible next subgoals, and recovery options. This work achieves good performance on SPA-Bench and AndroidWorld.

Critical strengths

This work successfully decouples the logical structure of a task from interface-specific UI elements, effectively coping with changes in interface layout.

It achieves significant improvements on both mainstream benchmarks (SPA-Bench and AndroidWorld)

As a built-in, IPS completes closed-loop mechanism spanning state matching to error recovery , which improves fault tolerance.

Critical weaknesses

It requires humans to hand-write a template of 4-8 subgoals for each task family, which limits full automation and scalability.

Questions for authors

Whether it is possible to reduce the reliance on manually authored task-family templates in the future, or outline a feasible approach for generating such templates automatically?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.