acceptodds
Under review as a conference paper at ICLR 2027

GUI-SKE: Structured Knowledge Evolution for Non-Parametric GUI Agent Adaptation

Abstract

GUI agents powered by multimodal large language models show promise for mobile task automation, yet can struggle in unseen applications where navigation structures and element-level interaction behaviors differ from prior experience. Parametric adaptation incurs additional optimization costs, while acquiring environment-specific knowledge and refining it using execution feedback remain challenges for non-parametric approaches. We introduce GUI-SKE, a non-parametric Structured Knowledge Evolution framework that adapts GUI agents by constructing and refining external knowledge while keeping model parameters fixed. GUI-SKE performs two-level exploration to acquire GUI-specific interaction evidence at different granularities and organizes it into linked functional, transition, and operational knowledge components. It derives pseudo-tasks with verification requirements from this evidence and uses rollout feedback as weak supervision for knowledge refinement. Group-wise trajectory comparison combines task-success and component-specific process feedback to guide targeted edits to the corresponding knowledge entries. Experiments on AndroidWorld, SPA-Bench, and our proposed ProMobile-Bench show that GUI-SKE achieves higher task success rates than the baselines included in our comparisons.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI agents, GUI adaptation, external knowledge, knowledge evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UjQgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GUI-SKE, a training-free framework that improves GUI agents through environment exploration and external knowledge refinement. It collects page-level navigation and element-level interaction evidence, organizes this evidence into functional, transition, and operational knowledge, and refines the knowledge through group-wise trajectory comparison without updating model parameters.

Critical strengths
  1. The component ablations provide initial evidence that knowledge refinement and component-wise feedback improve performance within the proposed framework.
Critical weaknesses
  1. The framework closely builds on GUI-explorer's knowledge acquisition and Training-Free GRPO's experience refinement. The methodological advance provided by the proposed knowledge organization and component-wise editing over existing approaches is not sufficiently established.

  2. Tables 4–5 demonstrate the usefulness of feedback and knowledge, but do not isolate the benefits of component-wise editing over unified editing with the same information and comparable budgets. This leaves the central methodological contribution insufficiently supported.

Questions for authors
  1. What is the key methodological advance over combining the closest prior approaches, particularly in knowledge organization and editing?
  2. Do the ablations isolate the benefit of editing knowledge by component from that of additional feedback? How are feedback information and computational budgets controlled?
  3. Did the authors check the final rendering of Figure 2 before submission? Please explain whether the apparent symbol rendering issues arose from an incorrect version, PDF export, or another cause.
Reviewer_8TePOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GUI-SKE, which adapts GUI agents to unfamiliar mobile apps by building and revising external knowledge while keeping model parameters frozen. It first collects interaction evidence through global and local exploration and organizes it into linked functional, transition and operational knowledge. It then generates pseudo-tasks with verification conditions from the exploration results and uses group-wise trajectory comparison with component-level feedback to make targeted edits to the relevant knowledge entries. On AndroidWorld, SPA-Bench and the authors' ProMobile-Bench, GUI-SKE achieves higher task success than the compared methods.

Critical strengths

The ablations are thorough and the gains point the same way on all three benchmarks. The improvements are 4.9, 3.0 and 5.4 points on AndroidWorld, SPA-Bench and ProMobile-Bench, and GUI-explorer on the last one was run by the authors. Table 4 breaks the 11.8-point total down by component, Table 13 shows that generating pseudo-tasks from exploration evidence is 7.5 points better than using the static prior alone, and E.4 shows that the Activity prior is worth 5.7 points for exploration. Together these indicate that the gain comes mainly from the update stage and not only from construction, which matches the authors' account.

The implementation and costs are reported openly, including prompts, output schemas, per-app token and API cost, human checks of pseudo-tasks, and per-step inference overhead. ProMobile-Bench uses two-stage verification, pulling the exported file through ADB and comparing it against a reference, which makes the evaluation fairly reliable.

Critical weaknesses

No variance is reported across the three reruns, and the main baseline numbers on AndroidWorld and SPA-Bench are copied from other papers rather than reproduced under the same setup. AndroidWorld results are sensitive to the GPT-4o version, the SoM implementation, prompts and emulator state, so a 4.9-point gap across papers is not very convincing. The M3A reproduction matches the original paper, but GUI-explorer was not rerun in the same codebase. Swings of about three points are common on 116 tasks, and without standard deviations it is hard to tell whether the main result is significant. ProMobile-Bench does use self-run baselines, but the gap between 20.3 and 25.7 amounts to five or six tasks out of 105 and needs variance as well.

The whole loop relies on the same model to judge and edit its own work, the reliability of the evaluators is not checked, and the structure emphasized in the title has no direct control. The paper does not test how much worse the update would be if the same exploration evidence were stored as unstructured text and refined with the same pseudo-tasks and group comparison, so the motivation that targeted component-level edits beat rewriting the whole thing lacks direct evidence. In addition, pseudo-task generation, success judgment, process evaluation, summarization, comparison and editing are all done by GPT-4o. Only the pseudo-tasks were checked by humans, and agreement between the success verifier or component evaluators and human judgment is not reported. Since the reward comes entirely from these evaluators, if they are inaccurate the knowledge updates may end up fitting the evaluators rather than real feedback from the environment.

Questions for authors
  1. Were the ablations in Tables 4 and 5 run once or several times? Can you give per-run results or standard deviations for the full evaluations on all three benchmarks?
  2. Can GUI-explorer be rerun with exactly the same code, GPT-4o version and emulator configuration as GUI-SKE? If not, please list the differences in model version, SoM annotation, maximum steps and similar settings.
  3. Please add an unstructured-knowledge control that stores the same exploration evidence as flat text and rewrites it as a whole, keeping everything else unchanged.
  4. What is the agreement rate with human labels for the success verifier and for each of the three component evaluators? The false positive rate of the success judgment matters most, because it directly decides the high and low reward groups.
  5. Many AndroidWorld tasks involve several functions or apps, but inference retrieves a single fine-grained function at the start. How does GUI-SKE do relative to the baselines on these tasks, and are there cases where retrieving the wrong function hurts performance?
  6. In Table 6 only this benchmark is marked as having Pro-Domain Tasks. What criterion defines this column? Which apps are in ProMobile-Bench, and will the tasks and verification scripts be released?
  7. The pseudo-tasks come from exploring the same apps that are evaluated. Did you check how much they overlap semantically with the 116 AndroidWorld task templates?
Reviewer_kwB8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

GUI-SKE is a non-parametric framework designed to enable GUI agents to adapt to unfamiliar mobile applications. The framework integrates both page-level and element-level exploration to construct an interconnected knowledge base of functionalities, transitions, and operations. The resulting knowledge base serves to guide agents as they execute tasks. Experimental results demonstrate that the framework improves task success rates.

Critical strengths
  • Well-motivated knowledge decomposition. A key feature of GUI-SKE is its organization of application-specific knowledge into three interconnected parts: functional, transition, and operational.
  • Exploration-based "pseudo-tasks" for knowledge refinement. This work uses collected interactions to automatically generate "pseudo-tasks" along with corresponding verification criteria. By executing these tasks and comparing the resulting interaction traces, the system obtains feedback that enables the refinement and enhancement of the knowledge base.
Critical weaknesses
  • Limited validation with stronger, more recent backbone models. The main experiments rely on GPT-4o, with an additional evaluation using Gemini 2.5 Flash only on AndroidWorld. It remains unclear whether GUI-SKE provides comparable benefits with stronger, more recent models, which may already possess better GUI understanding and navigation capabilities.
  • Insufficient controlled comparisons with related memory-based agents. There are many memory-based methods proposed for GUI agents for app adaptation, such as AppAgent[1] and MAGNET[2]. Authors should provide a controlled evaluation by comparing with these competing memory-based approaches under matched experimental settings and adaptation budgets. These comparisons are necessary to establish the advantage of the proposed functional–transition–operational organization over alternative memory designs.

[1] Zhang, Chi, et al. "Appagent: Multimodal agents as smartphone users." Proceedings of the 2025 CHI conference on human factors in computing systems. 2025.

[2] Sun, Libo, et al. "Magnet: Towards adaptive gui agents with memory-driven knowledge evolution." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors

Certain navigation paths may hinge on user permissions, account status, or personalized interfaces, while frequent application updates can render previously acquired knowledge obsolete. Given that GUI-SKE performs the adaptation process only once per application—with the resulting knowledge remaining static during subsequent executions—under what conditions can this knowledge be reused across different users or various versions of an application? Furthermore, how does the system detect entries that have become outdated or are no longer applicable? Does this necessitate a complete re-adaptation, or is it possible to perform selective updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.