acceptodds
Under review as a conference paper at ICLR 2027

TRACK: Autonomous Exploration and Task-Progress-Conditioned Knowledge Utilization for Mobile GUI Agent

Abstract

Offline knowledge acquisition and online knowledge utilization pipeline provides application-specific interaction knowledge for long-horizon mobile GUI tasks. However, existing methods do not sufficiently align how knowledge is acquired and reused with the evolving information needs of long-horizon GUI decision-making, often resulting in incomplete guidance or knowledge that is mismatched to the current task progress. Hence, we propose TRACK, a GUI Agent framework that constructs a multi-granularity knowledge base through Offline Exploration and Knowledge Synthesis, and adaptively injects the knowledge required at each step through Task-Progress-Conditioned Knowledge Utilization. During the offline stage, TRACK performs traceable and verifiable autonomous branch exploration around the major functionalities of each application. It then synthesizes the resulting exploration branches into complementary task-path knowledge for what to do, element knowledge for where to act, and skill knowledge for how to execute, forming a multi-granularity knowledge base. During online execution, we use the knowledge base to locate the current task progress from historical executions and to route the knowledge applicable to each decision step. By coupling multi-granularity knowledge base with offline autonomous exploration and progress-conditioned online utilization, TRACK provides complementary and progress-aligned guidance throughout long-horizon interactions. Experiments on AndroidWorld and DroidTask show that TRACK achieves strong task completion performance with marginally higher execution efficiency on Qwen3.5-Plus, while its constructed knowledge base further improves the success rates of integrated agents by up to 11.63% and simultaneously reduces their execution steps.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI Agent, Knowledge Synthesis, Retrieval-Augmented Generation, Offline Exploration, Knowledge Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bcwjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACK is a framework for acquiring and using application knowledge in mobile GUI agents. It explores applications offline using functional anchors, depth-first search, and transition verification, then produces task paths, element knowledge, and reusable skills.

During execution, this method localizes progress within retrieved paths and selects knowledge for the current decision.

The paper evaluates TRACK on 129 selected tasks from AndroidWorld and DroidTask with two decision models, and reports comparisons, ablations, and integration into other agents.

Critical strengths
  1. The paper addresses a practical limitation of retrieval for GUI agents: relevant knowledge can still suggest an inappropriate action, when prerequisites have not been met or earlier steps have already been completed. Linking knowledge use to execution progress is a reasonable way to address this problem.

  2. The results provide useful evidence that selective knowledge use can improve execution. On AndroidWorld, this method increases the GLM success rate from 65.12% without the knowledge base to 83.72%. It also outperforms simple similarity-based injection with the same frozen knowledge base on both models. These comparisons support the value of selecting knowledge rather than injecting retrieved records indiscriminately.

Critical weaknesses
  1. The contribution beyond existing methods is not sufficiently established.

In previous works, GUI-explorer[a] already combines functional anchors, DFS exploration, state restoration, and transition knowledge. Mobile-Agent-RAG[b] retrieves planning and operational knowledge at different levels, with operational retrieval conditioned on the current subtask. TRACK's main distinction therefore appears to be explicit path progress localization and conditional use of three knowledge types. However, Equation (4) and Appendix A.3 do not specify how the compatibility scores, weights, and localization confidence are computed. The supplementary reference code implements ranking and fixed truncation, but does not show the proposed joint localization and conditional routing. Table 3 also removes path knowledge and localization together. The current evidence does not separate the proposed progress mechanism from the benefits of filtering or reducing retrieved context. [a] Xie B, Shao R, Chen G, et al. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 5650-5667. [b] Zhou Y, Li J, Zhang Y, et al. Mobile-agent-rag: driving smart multi-agent coordination with contextual knowledge empowerment for long-horizon mobile automation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(35): 29939-29947.

  1. The evaluation does not yet provide sufficiently reliable evidence for the broader claims.

Section 4.2 states that DroidTask completion depends on the agent's self-judgment, while the experimental settings state that the original evaluators are retained. The original AutoDroid evaluation measures completion against annotated action sequences. It is unclear whether self-judgment only determines termination or also determines the reported success rate. The latter would make actual completion difficult to assess independently. AndroidWorld provides clearer evidence, but includes only 43 tasks. TRACK exceeds GUI-explorer by one successful task with Qwen and three with GLM, without reported variability across runs. These results suggest potential benefits, but do not establish a stable advantage that resolves the uncertainty about the main contribution.

Questions for authors
  1. Please clarify how the compatibility terms and weights in Equation (4), the localization confidence threshold, and the semantic step completion checks are implemented. Currently it seems not clear how does the supplied reference code relate to the implementation used for the experiments?
  2. Does the agent's self-judgment in DroidTask determine only when execution stops, or also whether a task is counted as successful? How is actual completion verified independently, and how does this procedure relate to the original evaluator?
  3. To validate the main distinction, can you compare progress localization with retrieval based on the current UI or subtask while retaining all three knowledge types, the same planner, and comparable context budgets? Limited repeated runs or paired task results for this comparison would help establish whether the benefit is stable. This request concerns validation of the present claims rather than an extension to new benchmarks.
Reviewer_apG7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACK, a mobile GUI-agent framework that combines offline autonomous application exploration with task-progress-conditioned online knowledge utilization. In the offline stage, TRACK extracts major application functionalities from the APK and uses them as exploration anchors, and then performs DFS-based exploration with state-aware backtracking and a Planner–Reflector–Recorder workflow to collect validated interaction branches. During online execution, it retrieves a relevant task path, estimates the current progress within that path, and selectively injects task-path, element, or skill knowledge according to the current decision state.

The method is evaluated on selected single-application tasks from AndroidWorld and DroidTask using two online decision models. TRACK achieves very good results on AndroidWorld and remains competitive on DroidTask.

Critical strengths

The paper presents a complete and well-engineered offline–online framework for application-specific GUI knowledge. The proposed TRACK combines autonomous exploration, validated trajectory collection, multi-granularity knowledge synthesis, and online knowledge use into a coherent pipeline, which is reasonable.

The progress-conditioned knowledge-utilization result is useful and supported by ablations. The experiments show that simply retrieving semantically similar knowledge can be ineffective or even harmful, whereas conditioning retrieval on the current task progress substantially improves performance.

Critical weaknesses

Most components of TRACK, including autonomous app exploration, structured navigation knowledge, reusable skills, retrieval augmentation, and state tracking, have been used in prior GUI-agent systems. The main contribution is the integration of multi-granularity knowledge with progress-conditioned routing. While this is reasonable, the contribution appears incremental rather than a substantial new algorithmic advance.

The paper relies on APK-derived application functionalities to bootstrap exploration and combines this with state-aware backtracking and multi-agent validation. However, this may not be a generalized method. And also, the transformation from raw APK artifacts into semantic application functionalities and exploration anchors is described only at a relatively high level.

Questions for authors

Please provide a clearer and more reproducible description of how application functionalities and exploration anchors are automatically extracted from the APK. The paper states that the APK pipeline analyzes the manifest, resources, and DEX and automatically produces an application-specific functional inventory without manual annotation. However, it is unclear how these low-level static artifacts are mapped to semantic functionalities.

Reviewer_vLr6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRACK, an offline-online framework for mobile GUI agents. The main idea is to first explore each app offline, build a multi-granularity knowledge base, and then use this knowledge during online task execution based on the agent’s current task progress. The knowledge base contains three types of knowledge: task paths for what to do, UI element knowledge for where to act, and skills for how to execute actions.

The key contribution is the task-progress-conditioned knowledge utilization mechanism, which tries to avoid injecting knowledge that is relevant to the task but not suitable for the current step. The paper also introduces an offline exploration pipeline with functional anchors, DFS-style exploration, backtracking, and knowledge synthesis.

The authors validate TRACK on AndroidWorld and DroidTask using Qwen3.5-Plus and GLM-5.3-Flash. The results show that TRACK improves task success rates over several GUI-agent baselines, especially on AndroidWorld. They also show that the constructed knowledge base can sometimes be plugged into other agents to improve performance.

Critical strengths

a) The paper targets an important problem in mobile GUI agents: how to reuse app-specific interaction knowledge for long-horizon tasks. The offline-online design is intuitive and practical, and the separation into task-path, element, and skill knowledge gives the agent complementary guidance on what to do, where to act, and how to execute.

b) TRACK shows clear performance gains on AndroidWorld and remains competitive on DroidTask across two different online models. The ablation studies support the value of the knowledge base, progress-conditioned routing, and different knowledge granularities. The plug-and-play experiments also suggest that the constructed knowledge base can help other agent frameworks in some settings.

Critical weaknesses

a) The paper combines several existing ideas—offline GUI exploration, retrieval-augmented decision making, trajectory/path knowledge, element grounding, and skill memory—into one framework. The progress-conditioned routing idea is useful, but the paper does not clearly show that it is a fundamentally new mechanism rather than a stronger engineering integration.

b) The experiments cover only 14 apps and 129 single-app tasks, and cross-app tasks are excluded. Since real mobile GUI use often involves more dynamic, personalized, and cross-application workflows, the current evaluation does not fully support the claim that TRACK generalizes to realistic long-horizon mobile automation.

c) TRACK depends on building an app-specific knowledge base before online execution, but the paper does not sufficiently report the offline exploration time, number of actions, LLM calls, token/API cost, or cost-benefit tradeoff. This is consequential because the practical value of the method depends heavily on whether the offline cost is reasonable compared with the online gains.

d) The system contains many modules, including APK-based functional anchors, DFS exploration, state-aware backtracking, multi-agent validation, knowledge synthesis, progress localization, and conditional routing. The ablations show that the knowledge base and routing help, but they do not isolate several key components, especially progress localization accuracy and offline exploration quality. As a result, it is hard to know which parts are truly responsible for the gains.

Questions for authors

a) How well does TRACK perform on unseen apps or on apps whose UI has changed after the knowledge base was built?

b) Were all baselines run with the same online model, action space, maximum step budget, UI information, and prompt-tuning effort as TRACK?

c) How much offline exploration is needed per app, in terms of wall-clock time, number of GUI actions, LLM calls, and token/API cost?

d) What is the performance impact of removing APK-based functional anchors, state-aware backtracking, or the Planner/Reflector/Recorder validation loop individually?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.