acceptodds
Under review as a conference paper at ICLR 2027

GCFSim: Learning Goal-Conditioned Functional Similarity for Mobile GUI Navigation

Abstract

GUI agents trained on large corpora of mobile-application trajectories can solve a wide range of user-requested tasks. A key challenge, however, is adapting such agents to a particular application without increasing inference latency. A promising approach is to explore the application offline, store application-specific knowledge, and retrieve relevant experience during navigation. This raises a fundamental question: which previously observed screen should the agent retrieve when deciding what to do next? We show that neither visual similarity nor goal progress alone is sufficient for this purpose. Two screens may look very different yet support equivalent useful actions for a particular task, while two states at the same distance from the goal may require different next actions. We formulate application-memory retrieval as goal-conditioned graph localization. Given the current screenshot and user goal, the objective is to retrieve a state from the explored application graph that provides useful guidance for the agent's next decision. For each goal, graph states are grouped according to the optimal actions they support and the goal-directed continuations reached through those actions. We use these functional relationships as supervision to learn GCFSim, a lightweight goal-conditioned visual-language representation . Although graph structure is used to construct training supervision, inference requires only the current screenshot and user goal. We evaluate GCFSim on a new cross-platform navigation corpus spanning 49 Android and HarmonyOS applications, with 7,903 observed UI states, 39,154 transitions, and 36,351 automatically generated navigation intents. The code and dataset are available anonymously for review and will be publicly released upon acceptance. Using the same application memories, functional supervision retrieves states that provide more useful navigation guidance than visual similarity, shortest-path distance supervision, and goal-conditioned value or Q representations. These results show that a useful retrieved state need not resemble the current screen or represent the same amount of progress toward the goal. What matters is whether the two states play the same functional role in completing the user's task.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GUI agents, mobile GUI navigation, goal-conditioned retrieval, application memory, functional similarity, graph localization, representation learning, state abstraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FWCLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles memory retrieval for GUI agents adapted to a specific application: given the current screenshot and a user goal, which previously observed state should be retrieved to guide the next action? The authors formalize this as goal-conditioned graph localization, define functional continuation classes — recursively, a state's optimal goal-directed transitions together with the continuation classes of their successors — and use these as supervision to distill a lightweight screenshot–goal representation (frozen unimodal encoders + a small projection). At inference, retrieval is a single cosine-similarity ranking with no graph traversal and no extra VLM call (29.6 ms, 205M params). A new corpus of 49 Android/HarmonyOS apps (7,903 states / 39,154 transitions / 36,351 intents) is built via VLM annotation with human auditing. GCFSim achieves the best NA@1 (.517) among nine baselines and consistently improves first-action selection for seven frozen GUI agents.

Critical strengths

S1: Application-memory retrieval is important for efficient GUI agents, and the distinction between visual similarity and functional similarity is well motivated.

S2: Motivation, design, and evaluation are coherently aligned: the paper's selling point is zero added inference latency, and it delivers exactly that (29.6 ms / 205M vs. 420 ms / 8.5B for LLM re-ranking). The paper includes multiple retrieval baselines, frozen-agent evaluations, continuation-depth analysis, robustness tests, and failure analysis.

S3: The experiments indicate that functional continuation supervision improves retrieval beyond visual similarity, goal distance, and immediate-action supervision.

Critical weaknesses

W1: The main metric and downstream experiments focus on first-action retrieval. The paper does not evaluate complete task success, long-horizon navigation, action efficiency, or recovery from mistakes.

W2: GCFSim is trained separately for each application, while training and test goals come from the same explored graph. Generalization to unseen applications, cross-platform transfer, and application-version changes is not demonstrated.

W3: The five-term loss (BCE + flip + sup + NCE + hard, weights 1.0/2.0/1.0/0.5/0.5) is never decomposed; no λ sensitivity.

W4: Notation and baseline citation issues. The Introduction introduces zθ(S,g)z_\theta(S,g) without defining (S)(S), (g)(g), or (θ)(\theta). In addition, the tables do not consistently identify which methods are prior work and which are author-implemented baselines. External models such as SigLIP, VisualGNN should have direct citations.

Questions for authors
  1. Complete task-level evaluation: Does retrieved functional memory improve complete task success, total interaction steps, execution time, or recovery after incorrect actions, rather than only first-action accuracy? Can you run even an end-to-end evaluation (e.g., AndroidWorld or MobileWorld) reporting full-task success with/without GCFSim memory?

  2. Generalization and data splits: Are the training, validation, and test goals strictly state-disjoint or path-disjoint within each application? Can GCFSim transfer to unseen applications, different platforms, or new application versions without training a separate model from scratch?

  3. Loss decomposition and sensitivity: What is the individual contribution of the five loss terms—BCE, flip, supervised contrastive, NCE, and hard-negative loss? How sensitive is performance to the coefficients (1.0/2.0/1.0/0.5/0.5), particularly the weight of the flip term?

  4. Fix the notation and citation problem.

Reviewer_oWQUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how mobile GUI agents can retrieve useful experience from previously explored applications. GCFSim uses the optimal actions available for a goal and their subsequent transitions to learn a representation of the screenshot and user goal. Experiments across 49 applications show improvements in retrieval and first-action selection for several frozen GUI agents.

Critical strengths

1.The retrieval problem is crucial to mobile GUI agents and memory reuse.

2.The controlled supervision study and continuation-depth ablation help explain where the gains come from.

3.The evaluation covers multiple applications and several GUI agents. The consistent improvements in first-action selection show that the retrieved guidance is useful across different model families/scales.

Critical weaknesses

1.The practicality of training a separate model for each application is not well established. Each application requires its own checkpoint, with no sharing of learned parameters across applications. Currently, the efficiency evaluation mainly measures online retrieval latency. It remains unclear how much offline work is required to support a new application and what needs to be repeated when an application changes.

2.NextAction@1 uses fixed rules to transfer actions within the explored graph, while the downstream experiments evaluate only the agent’s first action from a given screen. However, it remains unclear whether the gains persist over multiple steps, particularly when retrieved guidance is incorrect. Evaluation on end-to-end environments (e.g., MobileWorld, AndroidWorld), including task success and latency, would strengthen the practical motivation.

Questions for authors

Please refer to the weaknesses above. Two further concerns:

1.What are the training time and total offline preparation cost per application? When an application is updated, which parts of the graph construction, annotation, and training pipeline need to be repeated?

2.How to understand that the statement in the abstract that a useful retrieved state "need not represent the same amount of progress toward the goal", yet the proposed supervision relation treats all state pairs at different goal distances as negative pairs during training?

Reviewer_RbxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies memory retrieval for GUI agents: after offline exploration of an app, the agent retrieves, given the current screen and the user goal, the most useful historical screen to guide the current decision. The authors argue that neither visual similarity nor equal distance to the goal captures this, and propose measuring functional similarity between screens by whether they support equivalent optimal continuations under the current goal, and learning retrieval on this signal. On a self-built corpus of 49 Android/HarmonyOS apps, the method outperforms visual-similarity, distance-supervised, and graph-neural-network baselines, and improves first-action selection of several frozen GUI agents.

Critical strengths
  1. It identifies a real and overlooked problem. It separates "which historical screen to retrieve" from "retrieve the most similar screen"; the corpus covers 49 real commercial apps across two platforms, larger than most comparable work.
  2. Complete downstream validation and cost assessment. Multiple frozen agents plus an oracle upper bound validate downstream benefit, and inference cost is on par with the lightest baseline.
Critical weaknesses
  1. The evaluation protocol may overestimate retrieval quality; the claim that "inference needs only the screenshot and the goal" lacks direct evidence. A separate model is trained per app; the train/test split is over goals but shares the same state graph, so test-query screenshots likely appeared during training (paired with other goals), and training and test goals may point to the same pages. The paper never evaluates on screens that were entirely unseen during training, nor on any cross-app or cross-platform setting—which is exactly what would test the "lightweight representation, low-cost deployment" narrative. Moreover, downstream gains for most agents in Table 10 are only 0.6–1.7 points, with no significance testing.

  2. The relationship to state-abstraction theory, and the actual source of the gains, need more accurate positioning. The paper does cite the bisimulation line of work (Zhang et al., 2021; Hansen-Estruch et al., 2022) and compares against value/Q representations, which is commendable. However, Proposition 2 corresponds to known results of the form "the coarsest policy-preserving partition is bisimulation equivalence" (e.g., Li et al., 2006; Castro, 2020), and the closest work in spirit, the Policy Similarity Metric (Agarwal et al., ICLR 2021), is not discussed; the theoretical contribution should be framed accordingly. More importantly, the actual source of the gains: the authors' own ablation (Table 4) shows that grouping actions into just 2 coarse classes already reaches 0.513, nearly matching the full method's 0.517; and in the qualitative examples (Figs. 6, 8), the cases GCFSim gets right mostly rely on the two screens sharing the same persistent navigation bar or search button. In other words, what "functional equivalence" learns may be largely "which persistent control to tap and whether to scroll." A natural baseline—retrieval using "goal text + the current screen's UI-tree text"—would show what the learned embedding adds, but it is missing.

Questions for authors

Clarifications needed

  1. For each app, do test-query states also appear in that app's training graph (paired with training goals)? If so, could you run a simple experiment where some screens are entirely excluded from training and used only for testing?
  2. How were the 24 goals per app selected from the automatically generated intents? Do the target pages of training and test goals overlap?
  3. Roughly how well does a retrieval baseline using "goal text + the current screen's UI tree or page-description text" perform? How much would such a training-free baseline affect the conclusions?

Suggestions

  • Discuss the Policy Similarity Metric (Agarwal et al., 2021) in related work, and explicitly position Proposition 2 as an application of known bisimulation / policy-preserving abstraction results to the GUI setting.
  • Analyze what the two clusters in the K=2 condition of Table 4 are; quantify the proportion of judged-equivalent screens that share a persistent navigation control.
  • Add cross-app or cross-platform transfer experiments, as well as online multi-step task success rates.

Comments on the paper

1 comment
Area_Chair_GJjwOpenReview ↗

Reminder: Please Review Author Rebuttals and Participate in Discussions

Dear Reviewers,

Thank you for your time on the reviews. The authors have now submitted their rebuttals, and I encourage you to read their responses and participate in the discussion.

Please consider whether the rebuttals address your concerns and, where clarification is still needed, engage with the authors through the discussion forum. I also encourage you to exchange perspectives with your fellow reviewers and update your review or rating if appropriate.

Your participation will help ensure a thorough and fair evaluation of each paper. Thank you again for your time and effort.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.