- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes KnowAct-Memory, a test-time memory for mobile GUI agents. It turns past trajectories into parametric skills: JSON procedures with an action sequence, a visual precondition for each action, and fixed or dynamic slots (Sec. 3.2). At run time, a retrieved skill runs under state checks, with subgoal recovery when a precondition fails (Sec. 3.3). After each task, a Scriber agent distills successful traces, a Repair agent turns failed traces into candidate skills that need three real successes before promotion, and a Consolidation agent merges duplicates (Sec. 3.4). On AndroidWorld, MobileWorld and a 50-task HarmonyOS phone benchmark, with three open GUI backbones, per-step latency drops in every setting and success rises. For Qwen3-VL-8B on AndroidWorld, success goes from 45.68% to 52.58% and step time from 1.30 s to 0.97 s (Table 1).
Critical strengths
S1. A complete, well-explained design. The skill schema (Eq. 3, Table 10), the split between fixed and grounded actions, precondition checks with recovery, and the two-library lifecycle fit together well. Figure 2 and the prompts in App. D make the system easy to follow and to re-implement.
S2. Consistent efficiency gains. Per-step time falls in all seven comparisons in Table 1 (for example, -25% for Qwen3-VL-8B on AndroidWorld and -12% on MobileWorld), and step counts drop slightly in every row.
S3. Breadth and honest diagnostics. The real-device benchmark with a checked judge (Cohen's κ = 0.83 on 100 labels, Table 7) adds practical value. The ablation shows that naive skill memory hurts (GUI-Owl 60.34% to 53.45%, Table 4), and App. A.4 finds that overly coarse skills, not wrong retrieval, cause most retrieval-related failures (21 of 23). These are useful lessons for anyone building agent memory.
Critical weaknesses
W1. Most success gains are within run-to-run noise. Table 3 shows a spread of about 2 points between runs. The three-run gains there (+1.72 and +2.85 points) give t ≈ 1.2 and 1.6 over three runs, so they are not significant. Most other comparisons are single runs: the headline Qwen3-VL-8B gain (53 to 61 of 116 tasks), the margins over AWM and ReasoningBank (5 and 7 tasks, Table 2), each credited ablation step (4-6 tasks, Table 4), and the HarmonyOS gains (2 of 50 tasks). The same no-memory GUI-Owl setup scores 60.34% (Table 4), 61.20% (Table 1) and 58.91 ± 2.17% (Table 3), so a single run can move by 3-4 points. This matters because the abstract (L021-023) and the conclusion (L481-482) claim higher success. The evidence firmly supports lower latency at similar success; higher success needs repeated runs or narrower claims.
W2. The memory-building protocol is not stated. For Tables 1-4, I could not find which tasks built the memory, in what order, over how many passes, or whether memory was reset between runs. Eq. 2 (L146-147) updates memory with a success signal r_k, and promotion needs three successes (Sec. 3.4). If r_k is AndroidWorld's programmatic checker (L721) and the evaluated tasks also built the memory, the gain mixes learning from test labels with self-task replay. The held-out check (L402-403) covers only 20 tasks (9 to 13 solved). Also, the 397B row of Table 5 (43.3 to 46.2) matches Table 3, which suggests a Qwen3.5-35B-A3B extractor rather than the GPT-4o default (L249), and L653-654 says that in the 35B setting both conditions keep "textual experience memory". This matters because it decides whether the results show transfer ("generalizable", L084) and whether the baselines had the same budget.
W3. Some numbers do not match the stated task counts. The 60.75% in Table 1 (GELab-Zero-4B, AndroidWorld) is not k/116 for any k (70/116 = 60.3%, 71/116 = 61.2%), and it is not an average over two to four runs either. The 45.25% in Table 4 (Qwen3-VL, SGE only) is not k/116 either. None of the four MobileWorld success rates in Table 1 equals k/117, and Sec. 4.1 does not give the task count for that table. Readers will wonder whether some cells mix runs or task sets.
Questions for authors
Q1. Can you report the Qwen3-VL-8B AndroidWorld rows of Tables 1, 2 and 4 over three runs, with mean ± SD and a paired per-task test?
Q2. For each table and figure, which tasks built the memory, in what order, with which extractor, and what is r_k? Did the Table 2 baselines get the same tasks and extractor? Does a library built on disjoint tasks or new task seeds still help?
Q3. How were 60.75% and 45.25% computed, and how many tasks and runs are behind the MobileWorld rows of Table 1?