KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
Abstract
OpenClaw has emerged as a leading agent framework for complex task automation, yet its variants face two core bottlenecks: insufficient cross-platform GUI interaction support and no built-in self-evolution mechanism. We propose the "**Know Deeply, Act Perfectly**" paradigm, in which accumulated human–machine interaction and task-running experience improve execution accuracy and efficiency. Based on this paradigm, we introduce **KnowAct-GUIClaw**, a Know–Route–Act–Reflect framework that pairs a host agent for knowledge-grounded, long-horizon task decomposition with a pluggable GUI subagent equipped with experience-attributable memory and a self-evolving skill library. User profiles and feedback guide task decomposition and tool calls, while validated skills enable fast-path execution. Experiments across Android, iOS, HarmonyOS, and Windows examine manipulation efficiency, accuracy, and cross-platform adaptability. On the 117-task MobileWorld GUI-Only benchmark, the Kimi-K2.6 configuration achieves 64.1% pass@1, exceeding the compared systems, including Seed-2.0-Pro and GPT-5.5. Memory and skills support gains of 8.5 percentage points with Kimi-K2.6 and 16.2 points with Qwen3.5-35B-A3B, the latter using Kimi-derived experience.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.