acceptodds
Under review as a conference paper at ICLR 2027

KnowAct-Memory: A Self-Evolving Skill Memory for Continuous Learning in Mobile Agents

Abstract

Mobile GUI agents powered by multimodal large language models have been used to automate complex tasks. However, existing agents typically suffer from memoryless planning and regenerate action sequences from scratch for every task. This inefficiency increases inference latency and token cost while limiting the ability to learn from experience. We propose KnowAct-Memory, a self-evolving framework that shifts mobile GUI agents from heavy online planning to efficient state-grounded execution. Instead of replaying raw trajectories, KnowAct-Memory abstracts interaction traces into structured parametric skills, merges and refines them through semantic consolidation, and maintains a dual library distilled from both successful executions and repaired failures. Experiments on AndroidWorld, MobileWorld, and real-device HarmonyOS evaluations show that KnowAct-Memory improves measured success rates while reducing inference cost. With Qwen3-VL-8B on AndroidWorld, it improves success from 45.68% to 52.58% and reduces per-step inference time by 25.4%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
mobile GUI agents, agent memory, test-time learning, self-evolving agents, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gUS5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KnowAct-Memory, a test-time memory for mobile GUI agents. It turns past trajectories into parametric skills: JSON procedures with an action sequence, a visual precondition for each action, and fixed or dynamic slots (Sec. 3.2). At run time, a retrieved skill runs under state checks, with subgoal recovery when a precondition fails (Sec. 3.3). After each task, a Scriber agent distills successful traces, a Repair agent turns failed traces into candidate skills that need three real successes before promotion, and a Consolidation agent merges duplicates (Sec. 3.4). On AndroidWorld, MobileWorld and a 50-task HarmonyOS phone benchmark, with three open GUI backbones, per-step latency drops in every setting and success rises. For Qwen3-VL-8B on AndroidWorld, success goes from 45.68% to 52.58% and step time from 1.30 s to 0.97 s (Table 1).

Critical strengths

S1. A complete, well-explained design. The skill schema (Eq. 3, Table 10), the split between fixed and grounded actions, precondition checks with recovery, and the two-library lifecycle fit together well. Figure 2 and the prompts in App. D make the system easy to follow and to re-implement.

S2. Consistent efficiency gains. Per-step time falls in all seven comparisons in Table 1 (for example, -25% for Qwen3-VL-8B on AndroidWorld and -12% on MobileWorld), and step counts drop slightly in every row.

S3. Breadth and honest diagnostics. The real-device benchmark with a checked judge (Cohen's κ = 0.83 on 100 labels, Table 7) adds practical value. The ablation shows that naive skill memory hurts (GUI-Owl 60.34% to 53.45%, Table 4), and App. A.4 finds that overly coarse skills, not wrong retrieval, cause most retrieval-related failures (21 of 23). These are useful lessons for anyone building agent memory.

Critical weaknesses

W1. Most success gains are within run-to-run noise. Table 3 shows a spread of about 2 points between runs. The three-run gains there (+1.72 and +2.85 points) give t ≈ 1.2 and 1.6 over three runs, so they are not significant. Most other comparisons are single runs: the headline Qwen3-VL-8B gain (53 to 61 of 116 tasks), the margins over AWM and ReasoningBank (5 and 7 tasks, Table 2), each credited ablation step (4-6 tasks, Table 4), and the HarmonyOS gains (2 of 50 tasks). The same no-memory GUI-Owl setup scores 60.34% (Table 4), 61.20% (Table 1) and 58.91 ± 2.17% (Table 3), so a single run can move by 3-4 points. This matters because the abstract (L021-023) and the conclusion (L481-482) claim higher success. The evidence firmly supports lower latency at similar success; higher success needs repeated runs or narrower claims.

W2. The memory-building protocol is not stated. For Tables 1-4, I could not find which tasks built the memory, in what order, over how many passes, or whether memory was reset between runs. Eq. 2 (L146-147) updates memory with a success signal r_k, and promotion needs three successes (Sec. 3.4). If r_k is AndroidWorld's programmatic checker (L721) and the evaluated tasks also built the memory, the gain mixes learning from test labels with self-task replay. The held-out check (L402-403) covers only 20 tasks (9 to 13 solved). Also, the 397B row of Table 5 (43.3 to 46.2) matches Table 3, which suggests a Qwen3.5-35B-A3B extractor rather than the GPT-4o default (L249), and L653-654 says that in the 35B setting both conditions keep "textual experience memory". This matters because it decides whether the results show transfer ("generalizable", L084) and whether the baselines had the same budget.

W3. Some numbers do not match the stated task counts. The 60.75% in Table 1 (GELab-Zero-4B, AndroidWorld) is not k/116 for any k (70/116 = 60.3%, 71/116 = 61.2%), and it is not an average over two to four runs either. The 45.25% in Table 4 (Qwen3-VL, SGE only) is not k/116 either. None of the four MobileWorld success rates in Table 1 equals k/117, and Sec. 4.1 does not give the task count for that table. Readers will wonder whether some cells mix runs or task sets.

Questions for authors

Q1. Can you report the Qwen3-VL-8B AndroidWorld rows of Tables 1, 2 and 4 over three runs, with mean ± SD and a paired per-task test?

Q2. For each table and figure, which tasks built the memory, in what order, with which extractor, and what is r_k? Did the Table 2 baselines get the same tasks and extractor? Does a library built on disjoint tasks or new task seeds still help?

Q3. How were 60.75% and 45.25% computed, and how many tasks and runs are behind the MobileWorld rows of Table 1?

Reviewer_kNJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article proposes KnowAct-Memory, which parameterizes the interaction trajectory of mobile agents into skills. It drives the reuse of these skills through visual preconditions, fixed action execution, and dynamic positioning, and continuously updates the memory based on successful and failed trajectories. The article was evaluated on AndroidWorld, MobileWorld and HarmonyOS, and reported the changes in task success rate and inference cost.

Critical strengths
  1. The problem has practical significance. The article attempts to address the repetitive overhead and waste of historical trajectory value caused by the lack of memory in GUI agents when performing a complete task, and it has a clear research motivation and application value.

  2. The experimental design of the article is well-structured. Multiple backbone models were used to conduct tests on three benchmarks, and various memory baselines and results from repeated runs were also reported. It accurately reported phenomena such as the reduction in performance due to simple techniques and the fact that partial fixes being successful does not equate to the success of the entire task. This also provided a new perspective for the development of this field.

Critical weaknesses
  1. The memory construction and evaluation protocols are unclear, making it difficult to accurately interpret the main results. Table 1-3 fails to adequately explain aspects such as whether the historical data of the evaluated task is accessible, the source of the initial memory, and the reset rules between repeated runs.

  2. The innovation and contribution are not sufficiently demonstrated. Regarding the trajectory abstraction of GUI agent into reusable skills, learning from successes and failures, and maintaining memory, there have been well-developed studies. The experiments in this article cannot prove any substantial differences from the existing methods. The visual pre-check, candidate skill promotion, and other module designs may constitute distinctions, but the existing ablation studies have not verified the benefits of these designs. The authors need to clearly identify which mechanisms are new designs and which ones are derived from cross-domain transfer and combinations of existing solutions.

Questions for authors

See Weaknesses

Reviewer_5DMkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KnowAct-Memory, a self-evolving skill memory framework for mobile GUI agents that aims to reduce repeated planning through skill reuse. It represents skills as parameterized action sequences with visual preconditions and introduces a state-grounded execution mechanism that checks each action's visual precondition against the current screen, grounds dynamic actions at runtime, and recovers when a precondition is not met. The framework also incorporates skill extraction, failure repair, and lifecycle management to continuously improve the skill library without updating the backbone model. The method is evaluated on AndroidWorld, MobileWorld, and a real-device HarmonyOS benchmark. On AndroidWorld, it improves Qwen3-VL-8B's success rate from 45.68% to 52.58% while reducing per-step inference time by 25.4%. Additional comparisons and analyses examine its performance against existing memory-based methods, the contributions of individual components, and its ability to improve through continued experience.

Critical strengths
  1. Clear motivation and practical design. Mobile agents often re-plan routine operations such as login, search, and navigation, while interface layouts, pop-ups, and coordinates vary across sessions. KnowAct-Memory addresses this challenge by attaching visual preconditions to individual skill actions and distinguishing fixed actions from those requiring runtime grounding. It also requires skills extracted from failed trajectories to pass repeated execution validation before promotion, helping mitigate unreliable skill reuse and automatically generated skills.
  2. Comprehensive and informative evaluation. The evaluation covers multiple backbones, simulated benchmarks, and real-device tasks, providing evidence of the framework's effectiveness across different settings. Beyond comparing overall performance with existing methods, the ablation and self-evolution studies offer insights into why the framework works, particularly the importance of skill library management and the potential for continued improvement through experience accumulation. The additional analyses of failure cases, extraction costs, and auxiliary-model sensitivity further strengthen the empirical evaluation.
Critical weaknesses
  1. Insufficient details on the execution procedure. Section 3.3 refers to a “lightweight grounding model” and a “constrained planner” without specifying which models are used or how they operate. In particular, the paper does not explain how the constrained planner derives a short action sequence from the current state to the required precondition. Although Figure 5 provides an illustrative example, the underlying mechanism remains insufficiently described, making this part of the method difficult to understand and reproduce.
  2. Unclear accounting of inference costs. The framework introduces visual precondition verification, LLM-based retrieval filtering, dynamic-action grounding, and recovery planning, but the paper does not clearly specify whether the reported per-step inference time and token counts include the costs of these operations. In particular, reporting only output-token counts does not reveal the cost of processing screenshots for visual verification. It is therefore difficult to assess whether the reported 25.4% reduction in per-step inference time reflects the overall inference cost of the proposed framework.
  3. The novelty of the claimed contribution needs to be better established. The paper emphasizes the integration of visual precondition checks, fixed and dynamic actions, and skill lifecycle management to enable reliable skill reuse under changing UI conditions. However, Mobile-Agent-E (Wang et al., 2025) has already explored reusable Shortcuts with explicit preconditions and self-evolution based on execution histories and errors. This closely related work is not discussed, making it difficult to assess how KnowAct-Memory differs from and advances beyond existing approaches. The authors should include a discussion of Mobile-Agent-E and clarify the novelty of their integrated design.
Questions for authors
  1. Which model is the "lightweight grounding model" used for dynamic actions (Section 3.3)?
  2. Table 10 describes fixed actions as "invariant clicks or swipes", but the extraction prompt only says "mark fixed elements". What criterion does the Scriber Agent use to decide that an action is fixed? And what fraction of executed skill steps are fixed versus dynamic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.