acceptodds
Under review as a conference paper at ICLR 2027

ATLASACT: RETRIEVAL-CONSTRAINED GROUNDING AND STRUCTURE-VALIDATED MEMORY FOR MOBILE GUI AGENTS

Abstract

Multimodal large language models have enabled GUI agents to perform complex mobile tasks, yet agents may identify the intended control correctly while still failing to localize it precisely, and recurring interactions incur repeated model decisions. We propose AtlasAct, a framework that uses accessibility (A11y) structure as a shared executable interface for action grounding and interaction reuse. First, for element-targeted actions, the Actor specifies a target semantically, and retrieval-constrained grounding resolves it against multimodal A11y-node representations. A confidence router directly executes the top-ranked node when retrieval confidence is high and invokes the Grounder only when disambiguation is needed, reducing unnecessary Grounder calls. Second, structure-validated memory maintains Tips as decision guidance and Shortcuts as reusable action sequences. Shortcuts bypass per-step Actor and Grounder calls, while A11y-based entry and execution checks return control to the Actor upon structural mismatch. Experiments on four mobile GUI benchmarks demonstrate improvements in grounding and end-to-end task execution. Memory ablations show that checked Shortcuts reduce average decision steps by 10.4% relative to Tips alone on a fixed task subset, without lowering the observed overall success rate. With Qwen3.8-27B and interaction memory, AtlasAct achieves success rates of 67.67% on AndroidWorld and 95.40% on AndroidDaily, outperforming all evaluated baselines on both benchmarks. Our code is available at https://anonymous.4open.science/r/AtlasAct-A003.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI Agents, Multimodal Large Language Models, GUI Grounding, Interaction Memory, Accessibility Tree

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RMVsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ATLASACT, a mobile GUI agent framework that uses A11y structure for both action grounding and interaction reuse. For pointer actions, the Actor specifies a semantic target, after which relevant A11y nodes are retrieved and either directly executed or passed to the Grounder according to retrieval confidence. The framework also maintains Tips and Shortcuts, with structural entry and per-step checks used to constrain Shortcut replay. Experiments on four benchmarks show substantial improvements in grounding accuracy and end-to-end task completion.

Critical strengths
  1. The motivation is reasonable. Separating semantic target prediction from coordinate localization and using the available A11y tree can reduce execution errors.

  2. The reported performance improvements are substantial. Considering both the main paper and appendix, the evaluation is also relatively detailed, covering matched-backbone comparisons, end-to-end execution, memory ablations, confidence thresholds, candidate representations, and Shortcut-related hyperparameters.

Critical weaknesses
  1. The overall method resembles an engineering pipeline, and none of its individual modules provides a particularly convincing methodological contribution. Multimodal node retrieval is largely based on embedding matching, the confidence router operates on standard retrieval statistics, Tips and Shortcuts are inherited from Mobile-Agent-E, and the structural checks are mostly deterministic rules over existing A11y metadata. Although their integration may be practically useful, the technical novelty appears limited.
  2. Access to an A11y tree fundamentally distinguishes one family of GUI agents from methods that must infer actions directly from screenshots. Therefore, improvements over direct-coordinate agents do not fully establish the contribution of the proposed retrieval mechanism. The paper includes an AppAgent-style dense SoM baseline, but lacks sufficiently direct comparisons with closely related A11y-based approaches, particularly RecAgent-style component recommendation and M3A-style A11y action selection. Without such comparisons, it is difficult to determine whether the gains arise from the proposed design or simply from filtering and matching over structured UI elements.
  3. There are also some minor formatting issues. Several citations that should be parenthetical are rendered as narrative citations, particularly in the opening paragraphs, suggesting inconsistent use of \cite and \citep.
Questions for authors
  1. Could the authors clarify the most outstanding contribution of ATLASACT relative to prior A11y-based action selection, task-conditioned UI component filtering, and Mobile-Agent-E’s Tips and Shortcuts?

  2. Could the authors compare retrieval-constrained grounding with a RecAgent-style component recommendation baseline under a controlled budget? This would help isolate whether the proposed multimodal retrieval and confidence routing provide benefits beyond existing semantic candidate filtering.

  3. Could the authors also provide a controlled comparison with M3A-style A11y action selection using the same backbone and action space?

1 comment
AuthorsOpenReview ↗

Response to Reviewer RMVs:

We thank the reviewer for the constructive feedback.

W1 / Q1: Core contribution.

Our contribution is using current A11y structure as a shared executable interface for semantic target resolution and validated routine reuse.

Relative to M3A-style full-page marking, compact K=5 reduces clutter around small/medium controls (Table 12); relative to RecAgent-style text-based candidate recommendation, multimodal retrieval supports nodes lacking text/content description—29.96% of AndroidControl candidates—and selective grounding directly executes Top-1 on 52%–57% of AndroidControl pointer steps. Relative to Mobile-Agent-E-style verification, structural checks require no verification model calls and use 16.6%–21.1% fewer total LLM tokens; strong-check aborts return control to the Actor within the same decision step, with eventual full task completion in 11/14 cases (78.6%) (Table 5(b); Appendix A.4, Table 8(a)).

With Actor action types and queries fixed, and the same Top-5 size, router, and Grounder, the 27B all-step EM results are (Appendix F, Table 13):

Retrieval backendSMAN-Simple EM (%)SMAN-Complex EM (%)
Text-only64.3852.03
Patch-only61.6446.95
Text + Patch66.9054.06

Verification uses the same 42 tasks, fixed pipeline and Shortcut library, with post-task reflection disabled (Table 5(b)):

Verification methodSR (%)Average total LLM tokens
Structural checks79.7676.35k
Mobile-Agent-E-style per-step VLM checks79.7696.79k
Mobile-Agent-E-style entry + per-step VLM checks73.8191.59k

W2 / Q2–Q3: Controlled comparisons with RecAgent and M3A.

We added RecAgent-style and M3A-style baselines using the same 1,070 AndroidControl steps, seven-action space, and evaluator. RecAgent is our reimplementation; M3A receives the screenshot, full-page SoM, and full A11y list. The 27B comparison uses the same backbone without additional VLM fine-tuning (Appendix H, Table 16):

BackboneSplitM3A EM (%)RecAgent EM (%)AtlasAct EM (%)
7BAC-low56.5463.2788.22
7BAC-high48.1349.6370.28
27BAC-low84.7782.4387.85
27BAC-high68.6066.2668.13

Corresponding average LLM input/output tokens per step, with embedding input tokens in parentheses, are (Appendix H, Table 16):

BackboneSplitM3A LLM (+ Emb.)RecAgent LLM (+ Emb.)AtlasAct LLM (+ Emb.)
7BAC-low14.05k (0)15.28k (1.30k)4.98k (7.22k)
7BAC-high14.04k (0)15.71k (1.41k)4.99k (7.21k)
27BAC-low12.63k (0)11.70k (1.41k)4.72k (7.21k)
27BAC-high12.64k (0)11.99k (1.41k)5.20k (7.20k)

Embeddings use a single forward pass without autoregressive decoding. Even when LLM and embedding tokens are added 1:1 as a raw token-count comparison, AtlasAct remains lower than both baselines in all four settings.

AtlasAct improves EM over RecAgent in all settings with 57%–68% fewer LLM tokens. Against M3A, it improves EM in three settings; on 27B AC-high, M3A is 0.47 pp better, while AtlasAct uses approximately 59% fewer LLM tokens.

The online comparison also uses the same 27B backbone and action space on 116 Seed 30 tasks, with cross-task memory disabled in both agents (Table 3):

AgentCross-task memoryAndroidWorld SR (%)
M3A-style baselineOff53.45
AtlasActOff59.91

This comparison supports an execution benefit beyond A11y access alone, without introducing a cross-task-memory difference.

W3: Citation formatting.

We have standardized supporting citations throughout the main text, including the opening paragraphs, using \citep, and corrected the surrounding punctuation.

These A11y-based comparisons and controlled mechanism tests support more accurate executable target resolution and effective structural replay validation with lower LLM usage.

Reviewer_J9fYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ATLASACT, a mobile GUI agent framework that uses accessibility information for both action grounding and interaction reuse. For grounding, an Actor specifies a semantic target, multimodal retrieval selects candidate accessibility nodes, and a confidence router either executes the top-ranked node or invokes a Grounder for disambiguation. For memory, the framework maintains Tips and reusable Shortcuts, with accessibility-based admission and replay checks. Evaluation covers AndroidControl, SMAN-Bench, AndroidWorld, and AndroidDaily. The paper reports improved grounding accuracy and task success, together with fewer Actor decisions when using interaction memory.

Critical strengths
  1. The grounding result comparison provide meaningful evidence for the approach. With the same 27B backbone with no fine-tuning, ATLASACT improves over dense SoM by 10.30 and 6.43 percentage points on the simple and complex single-path SMAN settings, respectively. Improvements also appear on AndroidControl and in end-to-end execution.

  2. The memory ablation identifies a consequential failure mode of action reuse. With Tips enabled, adding unchecked Shortcuts reduces AndroidWorld SR from 66.81% to 65.09%, and enabling checks raises it to 67.67%. This suggests that replay requires explicit safeguards. The implementation details also distinguish strong checks that interrupt execution from weaker signals that merely record suspicion.

Critical weaknesses
  1. The contribution of the added mechanisms is insufficiently isolated from simpler alternatives. The paper acknowledges prior work on task-conditioned candidate filtering and Tips/Shortcuts memory. Against this background, the dense-SoM baseline establishes the benefit of the overall grounding pipeline, but does not establish which design choices are necessary. Appendix B already shows that direct Top-1 retrieval accounts for most of the grounding performance, and additional Grounder calls can even reduce accuracy on SMAN-Complex. This makes a comparison against a simple routing rule at a matched invocation rate particularly important. Similarly, the memory ablation compares only structural checks against unchecked replay. How isolating their advantage over the model-based applicability/outcome checks is not discussed.

  2. The evidence for the incremental memory benefit and its practical efficiency remains limited. AndroidWorld uses one instance per template, and memory accumulates in application order. No repeated-run is reported. In Table 4, full memory exceeds Tips-only by just 0.86 percentage points, approximately 1 more across 116 tasks. The 10.4% reduction in decision steps is informative, but excludes atomic actions within Shortcuts and does not account for slot-filling, reflection, retrieval, or verification overhead. The paper appropriately describes this metric as a proxy for Actor inference. The overall computational benefit of the additional memory machinery is not demonstrated.

Questions for authors
  1. Mechanism attribution: With the Actor and queries held fixed, how does text-only node retrieval compare with text+patch retrieval? At the same Grounder invocation rate, how does the learned router compare with a simple score-gap rule? For memory, can the structural checks be compared with model-based checks within the same framework?

  2. Robustness of memory gains: Do the differences between Tips-only, unchecked Shortcuts, and checked Shortcuts persist across task-instance seeds and task-order permutations? Please also clarify whether all memory-enabled baselines receive the same post-task outcome feedback.

  3. Efficiency and execution budgets: What are the total model calls, tokens, and wall-clock time when including memory slot-filling and reflection, node embedding, and replay verification? Are executed atomic actions subject to the same interaction budget across methods, including actions inside Shortcuts?

  4. Evaluation clarification: Appendix G.1 describes a pointer-matching rule that replaces the AgentCPM evaluator. Were all AndroidControl rows, including UI-TARS, rescored with exactly the same matcher?

These questions concern the interpretation and reproducibility of the results, and could affect my final assessment.

3 comments
AuthorsOpenReview ↗

Response to Reviewer J9fY:

We thank the reviewer for the constructive feedback.

W1 / Q1: Mechanism attribution.

We fixed the Actor’s action types and target queries and varied the retrieval backend, keeping the Top-5 candidate size, router, and Grounder unchanged. The 27B all-step EM results (%) are (Appendix F, Table 13):

Retrieval backendAC-lowAC-highSMAN-SimpleSMAN-Complex
Text-only74.0252.9264.3852.03
Patch-only74.0255.1461.6446.95
Text + Patch87.8568.1366.9054.06

Multimodal retrieval consistently improves the resolution of identical Actor-specified targets. On AndroidControl, 29.96% of candidate nodes lack both text and content description, helping explain the larger benefit of visual patches (Appendix F).

For routing, entries report pointer EM (%) / Grounder invocation rate (%) (Appendix B, Table 9):

Routing rule7B AC-low7B AC-high27B AC-low27B AC-high
Direct Top-180.79 / 3.5468.99 / 3.1383.08 / 4.0970.07 / 4.11
Score-gap rule85.83 / 47.8273.54 / 47.3786.36 / 43.5273.76 / 42.70
Learned router86.10 / 47.8273.97 / 47.2386.63 / 43.5274.04 / 42.84

Residual Direct Top-1 calls handle screens without interactive accessibility nodes. Score-gap invocation rates match the learned router within 0.14 pp. On AndroidControl, learned routing provides a modest additional accuracy gain while avoiding approximately 52%–57% of Grounder calls relative to always invoking it.

In SMAN-Complex, 31.5% of pointer steps are scrolls targeting large containers with a median box area of 60.9% of the screen, where Grounder re-selection underperforms Top-1 retrieval; additional calls can therefore reduce accuracy (Appendix B). At τ = 0.99, skipping approximately 48%–55% of calls relative to always invoking the Grounder preserves high-confidence Top-1 choices, although lower invocation can still perform better on this split.

For verification, we fixed the remaining pipeline and Shortcut library across 42 tasks and disabled post-task reflection (Table 5(b)):

Verification methodSR (%)Average total LLM tokens
Structural checks79.7676.35k
Mobile-Agent-E-style per-step VLM checks79.7696.79k
Mobile-Agent-E-style entry + per-step VLM checks73.8191.59k

The VLM baselines correspond to Action Reflector and Experience + Action Reflector, respectively; the latter also replaces entry selection. Structural checks match the observed SR of per-step VLM verification, whose total LLM usage is 26.8% higher. This supports effective replay validation without additional model inference for checking.

AuthorsOpenReview ↗

Response to Reviewer J9fY:

W2 / Q2: Incremental value and practical efficiency of memory.

We agree that the original 0.86-point SR improvement alone provides limited evidence for the incremental value of Shortcuts. We therefore evaluated sensitivity to task instances/order and whether fewer decisions translate into net LLM-token savings.

We added a within-application shuffled order and a new task-instance seed, each covering all 116 AndroidWorld tasks (Appendix C, Table 10):

SettingNo memory SR (%)Tips only SR (%)Tips + Shortcuts w/o checks SR (%)Full memory SR (%)Step reduction vs. Tips
Seed 30, original order59.9166.8165.0967.6710.4%
Seed 30, shuffled within applications59.9167.2468.9769.837.5%
Seed 42, original order63.7969.4069.8370.699.3%

The feedback protocols differ between internal ablations and external baselines:

  • AtlasAct’s memory-enabled ablations in Tables 4 and 10 receive identical binary reflection labels derived from official scores: score 1 denotes success; scores 0 and 0.5 trigger failure-mode reflection (Sec. 4.3).
  • External memory baselines, including Mobile-Agent-E, Agent S2, and SE-GA, retain their original inference, self-evaluation, and reflection procedures; official outcome signals are not injected into their reflection (Sec. 5.1).

Thus, the internal memory ablations are feedback-controlled, while external memory comparisons retain their native feedback protocols. The fixed-library verification comparison above disables post-task reflection, so feedback differences do not affect that mechanism test.

Without cross-task memory, AtlasAct already achieves 59.91% SR, exceeding same-backbone M3A-27B (53.45%) and Qwen3.8-27B (54.31%), providing evidence of a benefit independent of cross-task reflection feedback (Table 3).

To determine whether decision savings survive memory overhead, we counted execution, reflection, and slot-filling tokens on the same fixed 69-task subset in the original setting (Table 4):

ConfigurationActor/Grounder tokensReflection + slot-filling tokensTotal LLM tokens
No memory74.56k074.56k
Tips only71.81k2.01k73.82k
Shortcuts without checks70.27k1.88k72.16k
Shortcuts with checks66.83k1.85k68.68k
Tips + Shortcuts without checks68.65k3.77k72.42k
Full memory65.65k3.86k69.52k

Full memory’s 3.86k memory-processing overhead is already included. Its total LLM usage is 5.8% lower than Tips alone and 6.8% lower than No memory. We report input/output tokens because API and local runtimes depend on queueing, network conditions, hardware, and system load; token savings quantify LLM consumption without implying equivalent wall-clock acceleration.

Across the tested settings, Full memory achieves higher observed SR than Tips alone while reducing decision steps. These additional settings do not establish multi-run statistical significance. The clearest demonstrated benefit is lower inference demand after accounting for memory maintenance costs, alongside modest positive SR differences.

Q3: Embeddings, model calls, and execution budgets.

For the full-memory Seed 30 run, average LLM usage is 100.49k tokens (across all 116 tasks including capped failures, distinct from the 69-task subset above; Appendix H, Table 17; Appendix G.2). Node embeddings average 16.63k input tokens over 113 available reference logs and involve separate forward-pass computation. Memory processing averages approximately 1.9 model calls per task; complete system-wide call counts have not been aggregated.

We also expanded Shortcuts into primitive device actions to audit interaction budgets (Appendix G.2):

SettingFull-credit tasks exceeding the primitive-action capReported SR (%)SR after conservative budget rescoring (%)
Seed 300 / 7867.6767.24
Seed 420 / 8270.6970.69

One Seed 30 task receiving 0.5 credit exceeds the cap; conservative rescoring assigns it zero credit. This is post-hoc rescoring, not a rerun. We acknowledge this exception; all full-credit successes in both settings remain within the primitive-action budget.

Q4: AndroidControl evaluation consistency.

Yes. As clarified in revised Appendix G.1, all AndroidControl rows, including UI-TARS-1.5 and OS-Atlas, use exactly the same matcher and 1,070-step evaluation set. After action-type agreement, a pointer matches if it lies inside the smallest accessibility box containing the gold point or any of the five original boxes nearest that point. No method uses 1.2× box enlargement or the 0.04 normalized-distance fallback.

AuthorsOpenReview ↗

Response to Reviewer J9fY (Part 3): Correction to Table 13 (Patch-only) and Router Clarification

We follow up briefly with a protocol correction to the AndroidControl Patch-only entries in Part 1 (W1 / Q1, Table 13) and a short clarification on Table 9, matching our updated PDF:

1. Correction to AndroidControl Patch-only entries (Table 13 & Table 5(a)). In Part 1, the 27B Patch-only entries for AndroidControl had not been evaluated under the strictly frozen Text+Patch Actor protocol (previously, 27B AC-low shared the same value 74.02 as Text-only, and 27B AC-high carried 55.14 from the standalone closed-loop Patch-only run in Table 14). We have now completed the strictly frozen-Actor Patch-only runs across all four AndroidControl settings (7B/27B × AC-low/AC-high) and updated Table 5(a) and Appendix F Table 13 in the revised PDF.

  • In the 27B table shown in Part 1, only the two AndroidControl cells change (74.02 → 67.48 on AC-low, 55.14 → 48.12 on AC-high), while SMAN-Simple (61.64) and SMAN-Complex (46.95) remain unchanged:
Retrieval backendAC-lowAC-highSMAN-SimpleSMAN-Complex
Exact label51.4035.7858.8047.80
Patch-only67.4848.1261.6446.95
Text-only74.0252.9264.3852.03
Text + Patch87.8568.1366.9054.06
  • In the full Table 13 and Table 5(a) of the revised PDF, the corresponding 7B AndroidControl Patch-only all-step EM cells are also updated (74.21 → 67.66 on 7B AC-low, 57.98 → 54.02 on 7B AC-high), alongside all four AndroidControl pointer-step EM entries in Table 5(a) (56.27 / 49.50 / 56.75 / 48.94 for 7B AC-low / 7B AC-high / 27B AC-low / 27B AC-high). Across all four AndroidControl settings, both pointer-step EM and all-step EM strictly follow Exact label < Patch-only < Text-only < Text+Patch.

2. Practical distinction between the learned router and score-gap (Table 9). As detailed in Appendix A.2 and Appendix B, the LightGBM classifier is trained once on a separate 500-screen corpus and deployed with a single pre-fixed inference threshold (τ = 0.99) across all 7B/27B AndroidControl and SMAN-Bench settings without per-split tuning, whereas the score-gap thresholds (0.160–0.167 in Appendix B) were tuned per split to match the invocation rate.

Reviewer_BWdMOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mobile GUI agent framework that uses accessibility information for both action grounding and interaction reuse. An Actor describes the intended target, multimodal retrieval selects candidate accessibility nodes, and a confidence router either executes the highest-ranked candidate or invokes a Grounder for disambiguation. The framework also maintains cross-task Tips and executable Shortcuts, with structural checks controlling Shortcut admission and replay. It reports improved action accuracy and task performance, including 67.67% on AndroidWorld and 95.40% on AndroidDaily. The main contribution is integrating semantic decision-making with explicit interface structure to improve grounding and reduce repeated model decisions.

Critical strengths
  1. The problem is relevant and well motivated. Distinguishing target identification from precise execution addresses a concrete limitation of GUI agents. Using current accessibility observations to support both grounding and reusable routines is a coherent systems design.

  2. The grounding results are promising. With the same 27B backbone and no additional VLM fine-tuning, AtlasAct improves single-path EM over dense SoM by 10.30 and 6.43 percentage points on SMAN-Simple and SMAN-Complex, respectively. Improvements also appear in the online evaluations.

  3. The memory ablations provide useful evidence. Table 4 separates Tips, Shortcuts, and checks. The finding that unchecked Shortcuts can reduce performance, while checked Shortcuts recover it, is informative for practical agent design.

Critical weaknesses
  1. The experiments do not fully isolate the proposed grounding mechanism. Matching the backbone does not hold the action-generation pipeline fixed. For example, Table 1 reports SMAN-Simple Type Match of 87.66 for dense SoM versus 93.24 for AtlasAct at 27B. Target selection alone cannot explain this change in action-type accuracy. A comparison using identical frozen Actor outputs would better isolate retrieval-constrained grounding.

  2. Task-conditioned candidate filtering is already central to RecAgent, and Tips and executable Shortcuts with preconditions are introduced in Mobile-Agent-E. However, the evaluation should more directly establish the added value of its particular retrieval, routing, and deterministic verification mechanisms—for example, against a comparable candidate-filtering method and model-based verification within the same agent. The current results establish a useful combined system more clearly than they establish the necessity of its individual design choices.

  3. The online results lack sufficient evidence of robustness. In Table 4, full memory exceeds Tips alone by 0.86 percentage points, approximately one aggregate task-score unit on AndroidWorld. These are encouraging observations, but repeated task instantiations and task-order variations are needed to establish reliable gains. Since memory depends on preceding tasks, order sensitivity is especially relevant.

  4. Other presentation issues. I find that the citations are appended to text without appropriate parentheses or punctuation, making references difficult to distinguish from the surrounding prose and interrupting the reading flow. Please use consistent parenthetical formatting for supporting citations.

Questions for authors
  1. With Actor outputs fixed, how much of the grounding improvement remains? How do exact-label matching, text-only retrieval, and text-plus-patch retrieval compare under that controlled setting?

  2. Do the Table 3–4 gains persist across multiple task instantiations and shuffled task orders? Please provide paired outcomes, particularly for full memory versus Tips alone and checked versus unchecked Shortcuts.

  3. How frequently do structural checks reject valid Shortcuts, admit incompatible states, or successfully recover from replay failures? Can entry checks and per-step checks be evaluated separately using the same stored Shortcuts?

6 comments
AuthorsOpenReview ↗

Response to Reviewer BWdM:

We thank the reviewer for the constructive feedback.

W1 / Q1: Isolating grounding from Actor prediction.

We agree that the original end-to-end gains include action-type differences. For the cited SMAN-Simple 27B case, conditional EM on each method’s type-correct predictions remains 71.75% vs. 64.57% (+7.18 pp) in favor of AtlasAct (Appendix E, Table 11).

For stricter control, we fixed the Actor’s action types and target queries across retrieval backends. The 27B all-step EM results (%) are (Appendix F, Table 13):

Retrieval backendAC-lowAC-highSMAN-SimpleSMAN-Complex
Exact label51.4035.7858.8047.80
Text-only74.0252.9264.3852.03
Patch-only74.0255.1461.6446.95
Text + Patch87.8568.1366.9054.06

The embedding-based variants share the same Top-5 candidate size, router, and Grounder. Exact label requires a unique normalized match without Grounder fallback. Corresponding AndroidControl pointer-step results are provided in Table 5(a).

We also re-ran Full-page SoM at native screenshot resolution with Actor outputs fixed and the same Grounder prompt. The pooled AndroidControl and SMAN results on pointer steps with valid ground-truth boxes are (Appendix E, Table 12):

BackboneEvaluated stepsFull-page SoM EM (%)Compact Top-5 EM (%)Gain (pp)
7B3,88864.7472.63+7.90
27B3,89971.0275.94+4.92

Gains use unrounded scores. The [0.1%, 2.0%) screen-area band accounts for 65.0% of valid-box steps and shows gains of +10.79 pp at 7B and +7.82 pp at 27B, consistent with reduced marker clutter and occlusion around small and medium controls.

These frozen-Actor results support more accurate resolution of identical semantic targets, independently of action-type prediction.

W2: Added value beyond prior designs.

We build on RecAgent’s candidate filtering and Mobile-Agent-E’s Tips/Shortcuts. The following comparisons evaluate our specific retrieval, routing, and verification mechanisms.

Matched-backbone comparisons against a RecAgent-style reimplementation use the same 1,070 AndroidControl steps, action space, and evaluator (Appendix H, Table 16):

BackboneSplitRecAgent EM (%)AtlasAct EM (%)Gain (pp)
7BAC-low63.2788.22+24.95
7BAC-high49.6370.28+20.65
27BAC-low82.4387.85+5.42
27BAC-high66.2668.13+1.87

The 27B AtlasAct values reuse the Text+Patch results in W1. Across these settings, AtlasAct also uses 57%–68% fewer LLM tokens than RecAgent.

For routing, we compared against a score-gap rule at matched invocation rates (Appendix B, Table 9):

SettingRouter calls (%)Router pointer EM (%)Score-gap pointer EM (%)Gain (pp)
7B AC-low47.8286.1085.83+0.27
7B AC-high47.2373.9773.54+0.43
27B AC-low43.5286.6386.36+0.27
27B AC-high42.8474.0473.76+0.28

Invocation rates match within 0.14 pp. The learned router provides a modest additional accuracy gain while avoiding approximately 52%–57% of Grounder calls relative to always invoking it.

For verification, we fixed the remaining pipeline and Shortcut library across 42 tasks and disabled post-task reflection (Table 5(b)):

Verification methodSR (%)LLM tokens/taskToken increase
Structural checks79.7676.35k—
Mobile-Agent-E-style per-step VLM checks79.7696.79k+26.8%
Mobile-Agent-E-style entry + per-step VLM checks73.8191.59k+20.0%

The VLM baselines correspond to Action Reflector and Experience + Action Reflector, respectively; the latter also replaces entry selection. Structural checks achieve equal or higher observed SR at lower LLM cost, supporting effective replay validation without additional model inference for checking.

AuthorsOpenReview ↗

Response to Reviewer BWdM:

W3 / Q2: Robustness and the practical benefit of Shortcuts.

We added a within-application shuffled order and a new task-instance seed, each covering all 116 AndroidWorld tasks. Entries report SR (%) / average decision steps (Appendix C, Table 10):

SettingNo memoryTips onlyTips + Shortcuts w/o checksFull memory
Seed 30, original order59.91 / 9.8866.81 / 9.0665.09 / 8.5467.67 / 8.12
Seed 30, shuffled within applications59.91 / 9.8867.24 / 8.8668.97 / 8.6569.83 / 8.19
Seed 42, original order63.79 / 9.0369.40 / 9.4269.83 / 8.9670.69 / 8.54

We also measured total LLM usage in the original setting on the same 69-task subset, including Actor/Grounder calls, reflection, and slot filling (Table 4):

ConfigurationAverage total LLM tokensReduction vs. Tips only
Tips only73.82k—
Full memory69.52k5.8%

Across the tested settings, checked Shortcuts reduce decision steps while achieving higher observed SR than Tips alone. Checks improve SR and reduce steps over unchecked Shortcuts in all three settings (+2.58 / +0.86 / +0.86 pp SR; −4.9% / −5.3% / −4.7% steps); the original setting also demonstrates net LLM-token savings after memory overhead.

W4: Citation formatting and readability.

Thank you for pointing this out. We have standardized supporting citations. We have also added mechanism breakdowns clarifying the roles of the Actor, retrieval, routing, and replay checks to improve readability.

Q3: Check diagnostics and separate contributions.

We analyzed 52 Shortcut invocations involving 42 tasks (Appendix A.4, Table 8(a)):

DiagnosticCountRate
Strong-check aborts14 / 52 invocations26.9%
Entry-check aborts2 / 52 invocations3.8%
Mid-replay aborts12 / 52 invocations23.1%
Full task completion following Actor fallback11 / 14 abort-associated cases78.6%

All strong-check aborts return control to the Actor for replanning. Using the same 42 tasks and stored Shortcut library, we separately evaluated entry and per-step checks (Appendix A.4, Table 8(b)):

Entry checksPer-step checksSR (%)
OffOff70.24
OnOff67.86
OffOn78.57
OnOn79.76

Per-step validation provides the main reliability gain; entry checks add a smaller benefit when combined with it. Together with recovery following aborts, this supports monitoring applicability throughout replay and returning control to the Actor when checks fail. False-rejection and false-admission rates require independently labeled state compatibility and cannot be inferred from these abort counts.

These controlled results support AtlasAct’s central contribution: using accessibility structure for accurate target resolution and efficient, recoverable routine reuse. The evidence demonstrates grounding gains with Actor outputs fixed, lower-cost verification with the Shortcut library fixed, and reduced decision overhead across the tested task orders and instances.

I thank the authors for the response and additional experiments.The frozen-Actor comparisons, structured baselines, fixed-library tests, and token accounting address my concerns about isolation. However, I still have three remaining concerns:

  1. For 27B AC-high, Table 5(a) reports higher pointer EM for Text-only than Patch-only (56.17 vs. 55.19), whereas Table 13 reports lower all-step EM (52.92 vs. 55.14). With Actor outputs and non-pointer predictions fixed, shouldn't these orderings agree?

  2. I have also noticed Reviewer J9fY concern, that although you clarify that AtlasAct receives official outcome labels for memory updates, but external memory baselines retain their own self-evaluation. This confounds comparisons of memory mechanisms. Can you explain this further?

  3. It is unclear whether the new comparisons provide enough evidence to demonstrate that the learned router’s added complexity, given its small improvement in pointer EM over the simple score-gap rule at matched invocation rates.

I will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank the reviewers for their careful reading and constructive suggestions. Our detailed responses provide additional clarifications, controlled mechanism experiments, matched-backbone baseline comparisons, and token/budget audits. All corresponding additions and refinements in the revised PDF are highlighted in blue. We summarize the main points below:

  1. Isolating retrieval-constrained grounding and routing (Reviewers BWdM, J9fY, RMVs). To decouple target localization from Actor action-type differences, we added frozen-Actor retrieval-backend ablations (Text+Patch vs. Text-only, Patch-only, and Exact label; Table 5(a), Appendix F Table 13), type-correct conditional EM breakdowns (Appendix E Table 11), and native-resolution area-binned comparisons between Full-page SoM and Compact Top-5 SoM (Appendix E Table 12). We also evaluated the learned confidence router against score-gap and Top-1 score heuristics at matched Grounder invocation rates (Appendix B Table 9), and clarified why avoiding unnecessary Grounder re-selection preserves Top-1 accuracy on large scrollable containers in SMAN-Complex (Appendix B).

  2. Controlled comparisons with structured A11y baselines (Reviewers BWdM, J9fY, RMVs). To isolate benefits beyond A11y access alone, we added matched-backbone comparisons against M3A-style and re-implemented RecAgent-style A11y baselines on AndroidControl (Appendix H Table 16), as well as M3A-27B on AndroidWorld (Table 3 for SR, and Appendix H Table 17 for decision steps and LLM/single-pass embedding token usage).

  3. Replay verification, memory robustness, and cost accounting (Reviewers BWdM, J9fY, RMVs). Holding the pipeline and Shortcut store fixed with post-task reflection disabled, we compared deterministic structural checks against Mobile-Agent-E-style VLM verification (Table 5(b)), and added white-box abort/recovery diagnostics alongside a 2×2 entry vs. per-step ablation (Appendix A.4 Table 8). We further tested memory robustness under shuffled task order (Seed 30) and a new task-instance seed (Seed 42) (Appendix C Table 10), reported per-task LLM token totals including reflection and slot filling (Table 4, with component breakdowns in our response to Reviewer J9fY), and verified primitive-action step budgets after macro unrolling (Appendix G.2).

  4. Protocol clarity and presentation (Reviewers BWdM, J9fY, RMVs). We clarified that the internal memory ablations share identical binary official outcome signals (Sec. 4.3) whereas external memory baselines retain their original inference procedures (Sec. 5.1), confirmed uniform unscaled point-in-box evaluation across all AndroidControl baselines (Appendix G.1), and standardized citation formatting (\citep) and mechanism explanations throughout the manuscript.

Overall, these controlled comparisons and ablations support the effectiveness of using A11y structure as a shared executable interface: compact multimodal grounding improves frozen-Actor localization on pooled AndroidControl and SMAN-Bench pointer steps (+7.90 / +4.92 pp over Full-page SoM at 7B/27B, Table 12), AtlasAct outperforms matched-backbone M3A-27B on AndroidWorld (67.67% vs. 53.45% SR with 55.5% fewer LLM tokens), and deterministic structural checks match per-step VLM verification SR without checking-model overhead (Table 5(b)). Relative to Tips alone, full memory cuts decision steps by 7.5%–10.4% across the tested orders and seeds (Appendix C Table 10) and lowers total LLM tokens by 5.8% on the original task order (Table 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.