acceptodds
Under review as a conference paper at ICLR 2027

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

Abstract

OpenClaw has emerged as a leading agent framework for complex task automation, yet its variants face two core bottlenecks: insufficient cross-platform GUI interaction support and no built-in self-evolution mechanism. We propose the "**Know Deeply, Act Perfectly**" paradigm, in which accumulated human–machine interaction and task-running experience improve execution accuracy and efficiency. Based on this paradigm, we introduce **KnowAct-GUIClaw**, a Know–Route–Act–Reflect framework that pairs a host agent for knowledge-grounded, long-horizon task decomposition with a pluggable GUI subagent equipped with experience-attributable memory and a self-evolving skill library. User profiles and feedback guide task decomposition and tool calls, while validated skills enable fast-path execution. Experiments across Android, iOS, HarmonyOS, and Windows examine manipulation efficiency, accuracy, and cross-platform adaptability. On the 117-task MobileWorld GUI-Only benchmark, the Kimi-K2.6 configuration achieves 64.1% pass@1, exceeding the compared systems, including Seed-2.0-Pro and GPT-5.5. Memory and skills support gains of 8.5 percentage points with Kimi-K2.6 and 16.2 points with Qwen3.5-35B-A3B, the latter using Kimi-derived experience.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI agents, agent memory, skill learning, long-horizon planning, self-evolving agents, multimodal agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UgxdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Presents KnowAct-GUICLAW: a host+sub-agent GUI agent framework with experience memory, self-evolving skills, and a blackboard mechanism for cross-app data transfer. Reports competitive MobileWorld results and cross-model experience transfer.

Critical strengths
  1. State-validated skill reuse + blackboard cross-app information transfer is a clean, practical fix for common GUI agent failure modes (information loss, hallucination).
  2. Cross-model experience transfer is a nice result with real-world deployment value.
Critical weaknesses
  1. Core claims around long-horizon tasks and self-evolving memory have no matching benchmark evidence — OSWorld 2.0 and MemGUI-Bench are entirely untested.
  2. Baselines are outdated. No same-base ablation to isolate how much gain comes from the framework vs. the base model itself.
  3. Coarse joint ablation and inconsistent cross-platform evaluation protocols limit how conclusive the results are.
Questions for authors
  1. Any plans to evaluate on long-horizon / memory-focused benchmarks like OSWorld 2.0 or MemGUI-Bench?
  2. Can you add a same-base ablation (executor only vs. full framework) and update comparisons to current SOTA models (Qwen3.8-Max, Kimi K3)?
  3. Can you split host orchestration and memory in the ablation to quantify their individual contributions?
Reviewer_VEWUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper creates an agent architecture for digital task automation that runs task decomposition, knowledge retrieval, invokes a GUI subagent, and generates skills. Using Qwen3.5 and Kimi K2.6 as the underlying planner models, the architecture shows significant improvement in MobileWorld GUI-Only and AndroidDaily compared to closed models.

Critical strengths

It shows that a modular architecture with open models and good engineering can outperform monolithic closed models.

Critical weaknesses

There is no novelty in this work. The framework "pairs a host agent for knowledge-grounded, long-horizon task decomposition with a pluggable GUI subagent equipped with experience attributable memory and a self-evolving skill library." All of these architectural components have appeared in earlier work, some even dating back two or three years, and some of them already are cited by this paper. Even the whole architecture with task decomposition, skills, reflect has appeared before, e.g. the Voyager and Agent S papers that have task decomposition, hierarchical planning, knowledge retrieval, skill update etc.

For example "KNOWACT-GUICLAW distills trajectories into parameterized, state-validated skills, letting the executor select a reusable action prefix as one decision." This is exactly the same as the Voyager paper. Same for "Reflection summarizes traces, distills success and failure lessons into retrievable memory, and refines skills", which has been done in numerous works before, including GUI agents, too many to cite here.

The main focus of the paper is stated as "Hence, a natural research question arises: how can we endow OpenClaw-style agents with the ability to efficiently interact with visual graphical environments?" This question has low value for research. It's an engineering exercise to extend those popular open-source agents with GUI control actions and visual grounding. Doing this extension produces no new knowledge for research.

In section 3.1, there is a claim "The host agent keeps the running session context and forwards only what a GUI task needs when it issues that task." The "only" part is a strong claim. You need to give evidence for this claim.

In Section 4.1, MobileWorld is claimed to have long tasks. The average is just 27 steps, which is not long by today's standards (e.g. OSWorld 2.0 takes humans 1hr to do).

Furthermore, the paper uses the 117-task GUI-Only subset. Then you are not testing the overall ability of the agent framework to handle the general distribution of tasks that include both GUI and non-GUI tasks. That contradicts your previously stated motivation of having GUI executor being a subagent. If you just test on GUI-only tasks, you're not evaluating the accuracy of the main planner at deciding between GUI versus non-GUI tasks.

Accuracy is evaluated using a VLM judge, but the judge alignment with human judgement is not evaluated so it's unclear whether we can trust the judge.

Evaluation using pass@1 is no longer enough. The research and industry standard now is to show pass^k to measure reliability.

There is a router: "The router distinguishes single-app tasks from cross-app workflows." You need to report the router's accuracy at classifying single versus multi-app tasks.

Minor

  1. The abstract says "accumulated human–machine interaction and task-running experience improve execution accuracy and efficiency." Using experience to improve accuracy and efficiency is too vague. Not clear what is new about this.
Questions for authors

None

Reviewer_RViXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a GUI-agent framework organized around a Know-Route-Act-Reflect loop. The system separates high-level orchestration from low-level GUI execution: a host agent performs task decomposition, routing, information transfer, and tool use, while a GUI subagent handles visual interaction. The main additions are persistent experience memory, a typed blackboard for cross-app information transfer, and a self-evolving skill library that extracts reusable action sequences from prior trajectories and validates them against the current UI state before reuse. The strongest reported configuration reaches 64.1% pass@1 on MobileWorld, and the paper reports improvements in both success rate and execution efficiency from memory/skill reuse.

Critical strengths

There are three strengths after carefully reading this paper:

  • The paper targets an important and practical problem, and the overall system design is coherent. The separation between a host responsible for long-horizon reasoning and a GUI executor responsible for grounded interaction makes sense for realistic personal assistants. In particular, the typed input/output contract and blackboard are a sensible way to avoid passing increasingly long and lossy free-text histories across applications.

  • The skill representation is more thoughtful than simple trajectory replay. I like that skills are parameterized and guarded by expected-state checks, rather than being blindly replayed. The repair mechanism for failed skills and the validation of intents against the actual target page are also practically important. This is probably the part of the framework I find most convincing technically.

  • The empirical performance is strong. Regardless of some attribution issues discussed below, the system clearly produces a capable agent. On MobileWorld, the 35B configuration improves substantially over the base executor, while the kimi-k2.6 configuration reaches 64.1% pass@1. The skill-enabled configurations also reduce GUI steps rather than simply buying accuracy with substantially longer trajectories.

Critical weaknesses

The main concerns during the whole review processing are:

  • I am not convinced that the current evaluation establishes genuine self-generalization rather than reuse of experience from the benchmark being evaluated. The paper says that the kimi-to-qwen configuration uses memory and skills distilled from kimi-k2.6 trajectories, and immediately states that all MobileWorld configurations use the same 117-task GUI-only subset. However, I could not find a description of a held-out task split, held-out applications, or another separation between the trajectories used to construct skills and the tasks on which the resulting system is evaluated. This distinction is essential for the central claim. If a skill or memory was extracted after attempting the same task, or a near-identical instance of it, improved performance on a subsequent run is useful evidence of experience reuse, but it is quite different from showing that the agent has learned something that transfers to unseen tasks.

  • The novelty seems to lie mainly in the integration, but the paper does not yet make that distinction sharp enough. Several individual ingredients have fairly close precedents: [1] already studies accumulated reusable skills for a self-improving computer agent, [2] distills successful and failed experiences into retrievable memory, [3] extracts and retrieves reusable knowledge for mobile GUI execution, [4] provides parameterized computer-use skills with retrieval and recovery and [5] explicitly studies GUI/shortcut hybrid execution. I do think KnowAct-GUIClaw combines these ideas in a useful way, particularly through the host/executor boundary, typed information transfer, and state-validated skill reuse. But for an ICLR paper, I would like a clearer argument and preferably experiments, showing which part of this combination produces qualitatively new capability beyond these existing systems.

  • The ablations do not isolate the contributions the paper claims to make. For the 35B executor, A->B simultaneously introduces a much larger Qwen3.5-397B host, orchestration/routing, and memory. B->C then introduces skills. For the 397B setting, configurations E/F additionally allow direct host participation. The paper itself acknowledges that host orchestration and memory are jointly ablated. As a result, the 24.8->34.5 gain cannot really be called a memory gain, and more generally it is hard to tell whether performance comes from the proposed memory mechanism, decomposition, additional model capacity, direct host reasoning, external tools, or some combination. Given that these components are the primary methodological contribution, I think cleaner ablations are important: host without memory, host+memory, +trajectory-derived skills, +shortcuts, and ideally an ablation of the routing mechanism.

  • Some benchmark comparisons are difficult to interpret as controlled comparisons. For MobileWorld, the host is explicitly allowed to call external tools such as web search, while the headline result is reported on the GUI-only subset and compared against public leaderboard systems. The paper should be very explicit about whether such non-GUI tools are permitted under the exact evaluation protocol used for all compared systems. The host-mediated example in the appendix demonstrates that web search can sometimes directly resolve information that would otherwise require GUI interaction. I have a similar concern with AndroidDaily. Table 4 compares against UI-TARS and Step-GUI, but the paper states that its own AndroidDaily evaluation is performed on iOS devices, with unavailable tasks removed in the “Resolved” score and a mixture of LLM and human judging. This is interesting as a portability experiment, but unless the baseline numbers were rerun under exactly the same iOS environment and evaluator, I do not think the table supports a direct performance comparison.

Refs:

  1. OS-Copilot: Towards Generalist Computer Agents with Self-Improvement.
  2. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory.
  3. LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark.
  4. CUA-Skill: Develop Skills for Computer Using Agent.
  5. MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents
Questions for authors

I have some minor questions about this paper:

  • Q1: What exactly is the provenance of the memory and skill libraries used for each MobileWorld result? In particular, were any memories or skills used to evaluate a task extracted from an earlier execution of that same task, or from another task with essentially the same workflow? If so, could you report a held-out evaluation in which all memories/skills are generated from disjoint tasks (ideally also a held-out-app setting)? The answer to this question would substantially affect my assessment of the self-evolution claim.

  • Q2: During the MobileWorld GUI-Only evaluation, which non-GUI tools could the host use? Was web search or another external information source available? If yes, were the systems in Table 1 evaluated with the same capability? A result with external host tools disabled would make the comparison substantially easier to interpret.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.