- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents ATLASACT, a mobile GUI agent framework that uses A11y structure for both action grounding and interaction reuse. For pointer actions, the Actor specifies a semantic target, after which relevant A11y nodes are retrieved and either directly executed or passed to the Grounder according to retrieval confidence. The framework also maintains Tips and Shortcuts, with structural entry and per-step checks used to constrain Shortcut replay. Experiments on four benchmarks show substantial improvements in grounding accuracy and end-to-end task completion.
Critical strengths
-
The motivation is reasonable. Separating semantic target prediction from coordinate localization and using the available A11y tree can reduce execution errors.
-
The reported performance improvements are substantial. Considering both the main paper and appendix, the evaluation is also relatively detailed, covering matched-backbone comparisons, end-to-end execution, memory ablations, confidence thresholds, candidate representations, and Shortcut-related hyperparameters.
Critical weaknesses
- The overall method resembles an engineering pipeline, and none of its individual modules provides a particularly convincing methodological contribution. Multimodal node retrieval is largely based on embedding matching, the confidence router operates on standard retrieval statistics, Tips and Shortcuts are inherited from Mobile-Agent-E, and the structural checks are mostly deterministic rules over existing A11y metadata. Although their integration may be practically useful, the technical novelty appears limited.
- Access to an A11y tree fundamentally distinguishes one family of GUI agents from methods that must infer actions directly from screenshots. Therefore, improvements over direct-coordinate agents do not fully establish the contribution of the proposed retrieval mechanism. The paper includes an AppAgent-style dense SoM baseline, but lacks sufficiently direct comparisons with closely related A11y-based approaches, particularly RecAgent-style component recommendation and M3A-style A11y action selection. Without such comparisons, it is difficult to determine whether the gains arise from the proposed design or simply from filtering and matching over structured UI elements.
- There are also some minor formatting issues. Several citations that should be parenthetical are rendered as narrative citations, particularly in the opening paragraphs, suggesting inconsistent use of \cite and \citep.
Questions for authors
-
Could the authors clarify the most outstanding contribution of ATLASACT relative to prior A11y-based action selection, task-conditioned UI component filtering, and Mobile-Agent-E’s Tips and Shortcuts?
-
Could the authors compare retrieval-constrained grounding with a RecAgent-style component recommendation baseline under a controlled budget? This would help isolate whether the proposed multimodal retrieval and confidence routing provide benefits beyond existing semantic candidate filtering.
-
Could the authors also provide a controlled comparison with M3A-style A11y action selection using the same backbone and action space?
Response to Reviewer RMVs:
We thank the reviewer for the constructive feedback.
W1 / Q1: Core contribution.
Our contribution is using current A11y structure as a shared executable interface for semantic target resolution and validated routine reuse.
Relative to M3A-style full-page marking, compact K=5 reduces clutter around small/medium controls (Table 12); relative to RecAgent-style text-based candidate recommendation, multimodal retrieval supports nodes lacking text/content description—29.96% of AndroidControl candidates—and selective grounding directly executes Top-1 on 52%–57% of AndroidControl pointer steps. Relative to Mobile-Agent-E-style verification, structural checks require no verification model calls and use 16.6%–21.1% fewer total LLM tokens; strong-check aborts return control to the Actor within the same decision step, with eventual full task completion in 11/14 cases (78.6%) (Table 5(b); Appendix A.4, Table 8(a)).
With Actor action types and queries fixed, and the same Top-5 size, router, and Grounder, the 27B all-step EM results are (Appendix F, Table 13):
Verification uses the same 42 tasks, fixed pipeline and Shortcut library, with post-task reflection disabled (Table 5(b)):
W2 / Q2–Q3: Controlled comparisons with RecAgent and M3A.
We added RecAgent-style and M3A-style baselines using the same 1,070 AndroidControl steps, seven-action space, and evaluator. RecAgent is our reimplementation; M3A receives the screenshot, full-page SoM, and full A11y list. The 27B comparison uses the same backbone without additional VLM fine-tuning (Appendix H, Table 16):
Corresponding average LLM input/output tokens per step, with embedding input tokens in parentheses, are (Appendix H, Table 16):
Embeddings use a single forward pass without autoregressive decoding. Even when LLM and embedding tokens are added 1:1 as a raw token-count comparison, AtlasAct remains lower than both baselines in all four settings.
AtlasAct improves EM over RecAgent in all settings with 57%–68% fewer LLM tokens. Against M3A, it improves EM in three settings; on 27B AC-high, M3A is 0.47 pp better, while AtlasAct uses approximately 59% fewer LLM tokens.
The online comparison also uses the same 27B backbone and action space on 116 Seed 30 tasks, with cross-task memory disabled in both agents (Table 3):
This comparison supports an execution benefit beyond A11y access alone, without introducing a cross-task-memory difference.
W3: Citation formatting.
We have standardized supporting citations throughout the main text, including the opening paragraphs, using
\citep, and corrected the surrounding punctuation.These A11y-based comparisons and controlled mechanism tests support more accurate executable target resolution and effective structural replay validation with lower LLM usage.