acceptodds
Under review as a conference paper at ICLR 2027

TAP-Agent: From One-Way Handoffs to Two-Sided Preparation for Device–Cloud GUI Agents

Abstract

Capable graphical user interface (GUI) agents increasingly rely on cloud-hosted multimodal large language models for high-level planning and lightweight on-device models for visual grounding and execution. However, existing device–cloud frameworks treat each cloud response as a rigid handoff from planning to execution. This serialization forces the device to ground targets only after a plan arrives while leaving its computation underutilized during cloud reasoning, resulting in inaccurate execution and increased interaction latency. To address these limitations, we propose **TAP-Agent** (**T**wo-sided **A**nticipatory **P**reparation), which prepares on-device computation on both sides of each cloud response. After a response arrives, plan-conditioned visual experts provide heterogeneous visual evidence, which is compiled into action-ready representations that associate semantic elements with screen geometry. While awaiting the next response, a compact experience buffer of verified interactions guides the device to anticipate likely operations and prefetch supporting visual evidence. An alignment mechanism reuses this computation when it is consistent with the returned plan, while UI-changing actions remain governed by confirmed instructions. Together, these mechanisms coordinate computation across time, improving execution reliability while shifting reusable device inference off the critical path. Experiments on AndroidWorld and OSWorld with multiple on-device executors show that TAP-Agent improves task success by **5.7%** and reduces end-to-end interaction latency by **10.4%** over state-of-the-art device–cloud baselines. The code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, Device-Cloud Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EEr5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TAP-Agent turns the one-way plan-to-execution handoff in device–cloud GUI agents into two-sided preparation. Post-response, the cloud also specifies which local visual experts to run, and their outputs are compiled into an element-level representation tying semantics to screen geometry — lifting the executor from coordinate prediction to element selection. Pre-response, the device anticipates the next goal from verified experience and prefetches a fully resolved action, reusing it only when a semantic matcher confirms agreement with the returned plan.

Critical strengths

The problem is accurately characterized, identifying the "waiting period" as a schedulable resource. The authors point out that the serial device-cloud handoff gives rise to two seemingly unrelated issues—inaccurate grounding after the response and idle computing power before it—which are fundamentally the same temporal scheduling problem. Addressing both phases surrounding the response in a unified manner offers a more structural approach than optimizing isolated points (such as upgrading the local model or reducing cloud latency) and explains why both latency and success rates can be improved simultaneously.

Critical weaknesses
  1. Narrow experimental comparison. The only external baseline is EcoAgent, plus a fully-cloud upper bound. AAPT — the closest related work, which likewise precomputes during idle periods — is cited in a single sentence but never evaluated. The generality claim thus rests on "4 executors × 2 platforms" rather than on comparison against competing mechanisms, leaving the incremental value of two-sided preparation over existing idle-time precomputation unquantified.

  2. Confounded headline result. Nearly all of the success gain comes from the post-response visual tools (+8.7 / +4.8 pt), while the pre-response prefetching the title emphasizes only cuts latency — yet Table 1 merges them into a single +5.7 pt. The cell that isolates the tools (tools-on / prefetch-off, i.e. "EcoAgent + tools") stays in the ablation rather than the main table.

  3. Lack of fine-grained validation and failure analysis at the mechanism level. Seven tools employed only a coarse-grained "fully enabled/fully disabled" ablation approach, making it impossible to determine which specific components contributed to the results or which were rarely triggered.

Questions for authors

Is there any "false hit" — a prefetched action whose goal did not match the returned plan, yet still passed the semantic gate and the fresh-screen check and was executed? What is the false-hit rate, and what were the consequences?

1 comment
AuthorsOpenReview ↗

Reponse To Reviewer EEr5

Thank you for your insightful review and for recognizing the value of treating cloud waiting time as a schedulable resource. Your recognition of our unified treatment of grounding and idle time reinforces the value of two-sided preparation for improving task success and reducing execution latency. We respond to your concerns below with additional comparisons, per-tool ablations, and an analysis of false hits.

W1: The incremental benefit over existing idle-time precomputation is unquantified. AAPT — the closest related work, which likewise precomputes during idle periods — is cited in a single sentence but never evaluated.

A1: Thank you for raising this point. The key difference from AAPT is that TAP-Agent conditions prefetching on the actual post-action screenshot and retrieved experience, rather than preparing branches before the UI transition occurs.

We have compared TAP-Agent with AAPT using UI-TARS and OS-Atlas on AndroidWorld. Table 6 shows higher prefetch hit rates and lower execution time with both executors. These results support the advantage of preparing candidates from observed UI states and experience, allowing more prefetched work to be reused. We have added this comparison and expanded the distinction from AAPT in the revised manuscript (Section 5.4, Table 7).

Table 6: Comparison with AAPT on AndroidWorld using two executors. Time is the mean successful-episode duration.

ExecutorMethodPrefetch hit rate (%)Time (s)
UI-TARSTAP-Agent62.0101.3
UI-TARSAAPT33.8114.9
OS-AtlasTAP-Agent58.4120.5
OS-AtlasAAPT32.1134.7

W2: The headline result conflates the contributions of tools and prefetching. The cell that isolates the tools stays in the ablation rather than the main table.

A2: Thank you for highlighting the distinct contributions of these components. Due to the length constraints of the main text, we have added a comparison of EcoAgent + tools (tools on, prefetching off), EcoAgent, and TAP-Agent to the revised manuscript (Appendix D.1, Table 10), with success rates and execution times presented together. The two modules are also isolated in the main-text ablation in Section 5.3 (Table 4).

We have also changed the title to “TAP-Agent: Visual Tool Orchestration and Action Prefetching for Device–Cloud GUI Agents” to reflect both contributions: visual-tool orchestration improves task success, while action prefetching reduces interaction latency.

W3: Seven tools employed only a coarse-grained "fully enabled/fully disabled" ablation approach, making it impossible to determine which specific components contributed to the results or which were rarely triggered.

A3: Thank you for your valuable suggestion. We have measured tool usage and conducted individual tool-removal ablations with UI-TARS on both benchmarks. Table 7 reports calls during one full benchmark run and SR after removing each tool. Full TAP-Agent achieves 39.7% on AndroidWorld and 34.1% on OSWorld.

OmniParser is used most frequently on both platforms, and removing it leads to the largest drop in SR. Tool usage also varies by platform: zooming accounts for 21.7% of tool calls on OSWorld versus 13.8% on AndroidWorld, and removing it lowers OSWorld SR by 3.2 percentage points, showing its value in desktop environments. We have included the tool usage statistics, individual-tool ablations, and this analysis in the revised manuscript (Section 5.3, Table 6).

Table 7: Tool usage and individual removal ablations. SR in %.

PlatformMetricOmniParserOCRZoom inVisual QAROI MaskingAnomaly DetectionScroll Mapping
AndroidWorldCalls496273189305231268
AndroidWorldSR w/o tool (%)33.635.334.535.337.139.737.1
OSWorldCalls137288697572822186230
OSWorldSR w/o tool (%)30.332.030.931.732.833.632.0

Q1: Are there false hits? What is their rate, and what happens afterward?

A4: Thank you for your insightful question. We have audited the matcher's paired goal instructions using GPT-6.1 on AndroidWorld with UI-TARS. Among 850 prefetching rounds, 527 were accepted as hits (62.0%). GPT-6.1 identified 8 accepted hits as semantically mismatched, giving a semantic false-hit rate of 8 / 527 = 1.52%.

These false hits led to states in which the current plan could not continue. TAP-Agent handles such states through cloud replanning, at the cost of additional time and steps. We have added this semantic audit to Appendix D.3 of the revised manuscript.

Reviewer_MQrAOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper improves GUI agents' device-cloud cooperation by preparing on-device computation both before and after the cloud model responds. After the cloud responds, the device uses local visual tools to turn raw pixels into selectable UI elements. While waiting for the cloud's next plan, the device uses past experience to guess the next step and run visual tools in advance, improving success rates by 5.7% and cutting latency by 10.4%.

Critical strengths
  • Using idle waiting time during cloud planning to pre-run visual tools on the device effectively hides local computation delay.
  • Turning pixel clicking into selecting detected UI elements makes small on-device models much less likely to misclick.
  • Pre-computed actions are never executed until confirmed by the cloud plan, preventing wrong guesses from messing up the screen.
  • Experiments on AndroidWorld and OSWorld across four model sizes (2B to 9B) show consistent gains in both accuracy and speed.
Critical weaknesses
  • Running the executor, observer, and seven visual tools together requires too much memory and power for real phones, and was only tested on an RTX 4090 GPU.
  • Starting the next cloud planning request before the current action finishes can fail if the UI loads slowly or pops up unexpected windows.
  • The experience matching module relies on simple English keyword overlap, which may break on non-English apps or unseen interfaces.
  • When the pre-fetched guess is wrong (about 45% of the time), the extra local computation is completely wasted.
Questions for authors

How much memory, power, and latency does this full stack of models and visual tools consume on an actual mobile chip instead of an RTX 4090?

2 comments
AuthorsOpenReview ↗

Reponse To Reviewer MQrA

Thank you for your positive assessment and for recognizing TAP-Agent's gains in accuracy and efficiency across platforms and model sizes. The element-based grounding and idle-time prefetching which you highlighted underscore TAP-Agent's ability to improve both execution reliability and efficiency while preserving cloud control. We respond below with resource measurements and clarifications of the execution and recovery mechanisms.

W1 & Q1: Running the executor, observer, and seven visual tools together requires too much memory and power for real 1phones, and was only tested on an RTX 4090 GPU.

A1: Thank you for raising this practical concern. Table 5 summarizes resource consumption on an RTX 4090 and a Snapdragon 8 Elite Gen 5, with 4-bit UI-TARS on the mobile platform. On RTX 4090, the visual tools use 84% less peak memory and 87% less power than the executor on average, with 80% lower latency. On Snapdragon, the corresponding memory and latency reductions are 75% and 80%, with similar power draw. These results suggest that the tools are comparatively lightweight in memory and execution time, while deployment feasibility depends primarily on the on-device model's own resource requirements. We have added the measurements for both hardware platforms and their analysis to the revised manuscript (Appendix D.2, Table 11).

Table 5: Resource consumption on RTX 4090 and Snapdragon 8 Elite Gen 5.

HardwareComponentPeak memory (GB)Latency (s)Power (W)
RTX 4090Executor (7B, BF16)19.573.5330
RTX 4090Observer (2B)4.821.1183
RTX 4090Visual tools (avg.)3.160.742
RTX 4090Full local stack23.895.5270
Snapdragon 8 Elite Gen 5Device model (7B, 4-bit)6.9511.796.8
Snapdragon 8 Elite Gen 5Observer (2B)2.543.526.8
Snapdragon 8 Elite Gen 5Visual tools (avg.)1.752.416.2
Snapdragon 8 Elite Gen 5Full local stack11.2418.036.7

W2: Early cloud planning can fail when the UI loads slowly or an unexpected window appears.

A2: Thank you for your insightful comment. TAP-Agent triggers cloud replanning to resume execution when slow loading or an unexpected window makes the current plan inapplicable. Our toolkit includes Anomaly Detection, which allows the on-device model to detect unexpected situations through visual question answering, further improving robustness. The anomaly-detection tool and runtime checks are described in Appendices A.1 and B.1 of the revised manuscript.

W3: English keyword matching may fail on non-English apps or unseen interfaces.

A3: We appreciate your constructive suggestion. Our experiments are conducted in AndroidWorld and OSWorld, which are both English-language environments. We have added a lightweight translation and text normalization module to improve cross-lingual experience retrieval. On unseen interfaces, missing relevant experience may cause a prefetch miss, but TAP-Agent still follows the confirmed cloud plan and accumulates experience for the new environment from subsequent verified interactions. We have added these clarifications to Appendix C.2 of the revised manuscript.

W4: When the pre-fetched guess is wrong (about 45% of the time), the extra local computation is completely wasted.

A4: Thank you for raising this concern about computation cost. We introduced the tunable candidate budget M specifically to let downstream users balance local computation cost against time savings. A smaller M limits speculative work, while a larger M increases the chance of a useful prefetch. We selected M = 3 as a practical trade-off: Figure 4 in Section 5.4 of the revised manuscript shows diminishing hit-rate gains beyond three candidates, while additional candidates require more computation.

Reviewer_MQrAOpenReview ↗

thanks for your response

i will maintain my original rating

Reviewer_pbHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TAP-Agent targets GUI agents that split work between a cloud planner and a lightweight on-device executor. Existing frameworks treat each cloud response as a rigid handoff from planning to execution, so the small on-device model must localize targets from raw pixels on its own and often errs, while sitting idle during cloud reasoning. TAP-Agent prepares on both sides of each response: after a plan arrives, the cloud also specifies which local visual tools to call (element detection, OCR, regional zooming, etc.), and their outputs are compiled into an element-level representation with coordinates so the executor selects elements instead of predicting pixels; while awaiting the next response, the device uses an experience buffer to anticipate the likely next step and completes its grounding in advance, then reuses the prefetched action if the returned plan matches semantically and otherwise falls back to the standard path. On AndroidWorld and OSWorld with four on-device executors, it improves task success by 5.7% and reduces end-to-end latency on successful episodes by 10.4% relative to EcoAgent.

Critical strengths

The most notable strength is the alignment-gated asynchronous prefetching design. It converts the idle interval during cloud reasoning into useful on-device work by completing tool calls, grounding, and coordinate resolution ahead of time, yet UI-changing actions are still executed only after cloud confirmation, so the mechanism is speculative computation rather than speculative execution and never transfers control from the cloud to the device. The claim is well supported: in the 2×2 ablation with visual tools enabled, prefetching cuts successful-episode time by 24.0% on AndroidWorld and 14.0% on OSWorld while success rate stays within ±1%, indicating that the semantic gate reliably keeps correct prefetched actions and rejects mismatched ones; the hit-rate analysis further justifies M=3 as a compact candidate budget. Overall it is a clean systems idea whose latency gains come with no visible cost in reliability.

Critical weaknesses

Weakness 1 The success rate gains come almost entirely from the visual tools, and that component offers limited novelty. The ablation shows that enabling the tools alone accounts for the full SR improvement (31.0 to 39.7 on AndroidWorld, 29.0 to 33.8 on OSWorld) while prefetching adds nothing to SR. Yet using OmniParser for element detection with element ids as action targets, OCR and regional zooming for finer perception, and VQA for action verification and anomaly detection all appear in prior work such as OmniParser, GUI-Eyes, DetToolChain, and Zhang et al. 2026, which the paper’s own related work lists. What is specific to this paper is only that the cloud planner specifies which tools to call and that outputs are compiled into a unified representation, but there is no ablation isolating this, for instance no comparison against a device that always runs a fixed OmniParser plus OCR pipeline without cloud specified tool selection. It therefore remains unclear how much the plan-conditioned design itself contributes versus simply wiring in existing tools.

Weakness 2 The novelty of the prefetching mechanism is asserted rather than demonstrated, and the comparison set is thin. The related work acknowledges that AAPT already precomputes conditional policy trees during idle periods and claims the difference lies in semantic alignment gating, but there is no empirical comparison with AAPT or any other precomputation or pipelined planning approach; overlapping the next request with the current action and gating the reuse of speculative results is also a well established speculative execution pattern in systems. The only baseline is EcoAgent reimplemented under the authors’ configuration, with no OpenPhone, MAI-UI, or other device and cloud collaborative methods, while the fully cloud reference remains far ahead. Combined with the latency protocol issues (failed episodes counted at the timeout value, different successful sets per method) and the appendix admission that experience reflection actually calls the cloud model, which conflicts with the on-device experience narrative, the attribution of the 10.4% latency gain to the proposed mechanism is not well grounded.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Reponse To Reviewer pbHk [1/2]

Thank you for your careful review and for recognizing the efficiency and reliability benefits of alignment-gated asynchronous prefetching. Your observations on preserving cloud control and speculative computation help us clarify the mechanism and strengthen the evidence for its benefits. We address each concern below with additional comparisons and methodological clarifications.

W1.1: How does the visual toolkit advance beyond existing perception tools?

A1: Thank you for this question. TAP-Agent introduces, to our knowledge, the first comprehensive, multi-category visual expert toolkit for device–cloud GUI agents. It combines OmniParser and OCR with custom Scroll Mapping and ROI Masking tools, organizing their outputs into a common representation for local execution.

Table 1 shows that removing OmniParser causes the largest drop in SR, while removing Scroll Mapping or ROI Masking also lowers SR by approximately 2 percentage points on average across the two benchmarks. These results support the advantage of our multi-tool library, which combines complementary capabilities for localization, visual inspection, state verification, and tracking across screen changes. We have added the combined comparison to Appendix D.4 (Table 12) and the complete individual-tool results and analysis to Section 5.3 (Table 6) of the revised manuscript.

Table 1: Toolkit comparison and individual tool ablations with UI-TARS. SR in %.

Method / configurationAndroidWorld SROSWorld SR
EcoAgent31.029.0
TAP-Agent w/o cloud-specified tool selection26.726.0
TAP-Agent39.734.1
TAP-Agent w/o OmniParser33.630.3
TAP-Agent w/o Visual QA35.331.7
TAP-Agent w/o ROI Masking37.132.8
TAP-Agent w/o Scroll Mapping37.132.0

W1.2: What does cloud-specified tool selection contribute beyond a fixed OmniParser + OCR pipeline?

A2: Thank you for this constructive question. Cloud-specified selection contributes task-conditioned evidence acquisition, rather than simply exposing the device model to more visual-tool outputs. For each subgoal, the cloud planner selects the relevant tools and specifies the evidence they should acquire.

Table 2 presents a 2×2 ablation of toolkit coverage and cloud-specified selection; disabling the full toolkit retains OmniParser + OCR. Without cloud guidance, the fixed pair achieves 29.3% / 30.9% SR, compared with EcoAgent's 31.0% / 29.0% (Table 1).

We additionally used GPT-6.1 to assess each tool-output entry passed to the device model for relevance to the task goal. In this audit, the unguided fixed pair produces 3,054 characters of tool-output text with 10.2% relevant entries, compared with 1,640 characters and 18.5% for TAP-Agent. These results show that cloud guidance makes a broader toolkit useful by selecting subgoal-relevant evidence and suppressing distracting outputs, thereby improving device-side execution. We have included these ablations and the input-relevance analysis in the revised manuscript (Section 5.3, Table 5).

Table 2: 2×2 ablation of the full toolkit and cloud-specified tool selection with UI-TARS. SR in %. Average input length and relevance refer to tool outputs passed to the device model; GPT-6.1 assesses relevance entry by entry. ✓: enabled; ✗: disabled.

Full toolkitCloud-specified tool selectionAndroidWorld SROSWorld SRTool-output length (chars)Relevant entries (%)
✗✗29.330.93,05410.2
✗✓36.232.81,57217.6
✓✗26.726.04,1298.3
✓✓39.734.11,64018.5
AuthorsOpenReview ↗

Reponse To Reviewer pbHk [2/2]

W2.1: There is no empirical comparison with AAPT or another precomputation approach.

A3: Thank you for this feedback. TAP-Agent differs from AAPT in when and from what state prefetch candidates are constructed. TAP-Agent initiates cloud planning while the current action is in flight, then uses the realized post-action screen and retrieved experience to prepare the next action during the remaining cloud-reasoning interval. In contrast, AAPT expands speculative branches before the resulting UI state is observed.

We have compared the two methods on AndroidWorld with UI-TARS and OS-Atlas (Table 3). TAP-Agent achieves prefetch hit rates of 62.0% / 58.4%, compared with 33.8% / 32.1% for AAPT, with lower execution time for both executors. These results support the advantage of preparing candidates from observed UI states and experience, allowing more prefetched work to be reused. We have included this comparison and analysis in the revised manuscript (Section 5.4, Table 7).

Table 3: Comparison with AAPT on AndroidWorld. Time is the mean successful-episode duration.

ExecutorMethodPrefetch hit rate (%)Time (s)
UI-TARSTAP-Agent62.0101.3
UI-TARSAAPT33.8114.9
OS-AtlasTAP-Agent58.4120.5
OS-AtlasAAPT32.1134.7

W2.2: The comparison omits MAI-UI and OpenPhone, while the fully cloud reference remains far ahead.

A4: Thank you for this constructive suggestion. Instead of transferring action generation to the cloud when local execution needs help in MAI-UI and OpenPhone, TAP-Agent uses cloud plans to guide device-side grounding.

We have added comparisons using UI-TARS and OS-Atlas on both benchmarks (Table 4). TAP-Agent has the lowest execution time and fewest cloud calls in all four configurations, and the highest or tied-highest SR in three. OpenPhone achieves a slightly higher SR in one configuration, with longer execution time and more cloud calls. We have included these comparisons and their analysis in the revised manuscript (Section 5.2, Table 3), while retaining the fully cloud reference in its Table 1.

Table 4: Comparison of collaboration strategies. SR in %, mean successful-episode time in seconds, and cloud calls in the full benchmark.

Method / strategyExecutorAndroidWorld SRAndroidWorld timeAndroidWorld cloud callOSWorld SROSWorld timeOSWorld cloud call
EcoAgentUI-TARS (7B)31.0128.5116429.0160.74879
EcoAgentOS-Atlas (4B)24.1136.9152519.9157.94968
MAI-UIUI-TARS (7B)34.5280.8197129.3321.87031
MAI-UIOS-Atlas (4B)25.9299.9207924.1390.07526
OpenPhoneUI-TARS (7B)34.5320.8204032.8340.47142
OpenPhoneOS-Atlas (4B)31.9378.4209217.2380.57269
TAP-AgentUI-TARS (7B)39.7101.395834.1142.34498
TAP-AgentOS-Atlas (4B)31.0120.5123524.1139.84596

W2.3: Failed episodes are counted at the timeout, and methods use different successful sets, making the attribution of the latency gain unclear.

A5: Thank you for raising this point. All-episode time assigns failed episodes the configured timeout, which is 450 s for the device–cloud methods and 1,200 s for the fully cloud reference. This metric therefore reflects both task completion and execution duration. Successful-episode time averages actual durations over each method's completed episodes, and these successful sets are method-specific.

To assess the contribution of prefetching with visual processing held fixed, the tools-enabled ablation compares prefetching off versus on (Section 5.3, Table 4). Successful-episode time decreases 24% on AndroidWorld and 14% on OSWorld, with nearly unchanged SR. This ablation supports the latency benefit of overlapping local preparation with cloud reasoning. We have clarified the metric interpretation and the distinct roles of tools and prefetching in the revised manuscript (Section 5.3).

W2.4: Experience reflection calls the cloud model, which conflicts with the on-device experience narrative.

A6: Sorry for the confusion. Experience reflection is generated through asynchronous cloud requests during the interaction loop, without blocking its execution. Experience retrieval and matching are performed locally on the device. We have clarified in the revised manuscript (Section 4.2 and Appendix C.1) that "on-device experience" refers to local storage and retrieval.

Thanks to the authors for the detailed response! It addressed several of my concerns: the missing baseline with a fixed OmniParser + OCR setup and no cloud-side tool selection, the comparison with AAPT, leaving out MAI-UI and OpenPhone, how latency is measured, and the experience reflection step calling the cloud.

However my main concern is still there: the novelty feels quite limited to me. Overall, the paper still reads more like solid engineering than a research contribution.

So while I really appreciate the thorough rebuttal, I’m going to keep my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Area Chair and Reviewers,

We thank all reviewers for their careful evaluation and constructive suggestions. We particularly appreciate their recognition of treating cloud waiting time as a schedulable resource (EEr5), alignment-gated prefetching that reduces latency while preserving cloud control (pbHk), and improvements across platforms and executor sizes (MQrA). These observations highlight the value of coordinating visual grounding and advance preparation to improve task completion and interaction efficiency.

Below, we clarify our core contributions and summarize the additional experiments and explanations included in the revised manuscript.

Contributions

TAP-Agent strengthens the local executor through two complementary mechanisms:

  1. A visual expert toolkit with cloud-specified invocation. Our multi-category toolkit combines various perception tools, providing complementary capabilities for localization, visual inspection, state verification, and tracking across screen changes. The cloud planner specifies which tools to invoke, whose outputs are compiled into an action-ready representation, giving the executor relevant visual evidence for element-level action selection.
  2. Experience-based, alignment-gated prefetching. During cloud reasoning, the device uses the actual post-action screenshot and retrieved experience to prepare likely next actions. These actions are reused after alignment with the returned cloud plan and screen-validity checks. This overlaps local preparation with cloud reasoning while retaining cloud confirmation for UI-changing actions.

Therefore, we have revised the title to “TAP-Agent: Visual Tool Orchestration and Action Prefetching for Device–Cloud GUI Agents” to reflect both contributions.

Additional experiments

Thanks to the reviewers' suggestions, we have added the following evidence to the revised manuscript:

  • Toolkit coverage and cloud guidance [pbHk, EEr5]. The 2×2 ablation and input-relevance audit show how cloud-specified selection helps the executor benefit from a richer toolkit. Tool-usage statistics and removal ablations reveal complementary and platform-dependent contributions (Section 5.3, Tables 5–6; Appendix D.4, Table 12).
  • Competing collaboration and precomputation methods [pbHk, EEr5]. We have added comparisons with MAI-UI and OpenPhone, where TAP-Agent achieves competitive SR with the lowest execution time and fewest cloud calls, and with AAPT, where TAP-Agent achieves higher prefetch hit rates and lower execution time (Section 5.2, Table 3; Section 5.4, Table 7).
  • Separating success gains from latency savings [pbHk, EEr5]. The added EcoAgent + tools comparison reports SR and execution time together, complementing the existing two-module ablation and making their distinct effects explicit (Appendix D.1, Table 10; Section 5.3, Table 4).
  • Deployment cost on mobile devices [MQrA]. We have added component-level and full-stack memory, latency, and power measurements on RTX 4090 and Snapdragon 8 Elite Gen 5, showing much lower average memory and latency costs for the tools than for the executor (Appendix D.2, Table 11).
  • False hits and recovery [EEr5]. We have added a semantic false-hit audit and described the resulting recovery through cloud replanning. Results prove the low false hit rate and robustness of the recovery mechanism against unexpected states (Appendix D.3).

Key clarifications

To address the reviewers' concerns, we have clarified below details for readability:

  • asynchronous cloud reflection versus local experience storage and retrieval (Section 4.2 and Appendix C.1)
  • translation and fallback on unseen interfaces (Appendix C.2)
  • the computation–latency trade-off controlled by M (Section 5.4)
  • timeout treatment and successful-episode timing protocol explicit (Sections 5.1–5.2)

We further reorganized the experimental analysis to connect each result to the contribution it evaluates. Thank you again for your constructive feedback. We look forward to further discussion and provide detailed answers in the reviewer-specific responses below.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.