- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TAP-Agent turns the one-way plan-to-execution handoff in device–cloud GUI agents into two-sided preparation. Post-response, the cloud also specifies which local visual experts to run, and their outputs are compiled into an element-level representation tying semantics to screen geometry — lifting the executor from coordinate prediction to element selection. Pre-response, the device anticipates the next goal from verified experience and prefetches a fully resolved action, reusing it only when a semantic matcher confirms agreement with the returned plan.
Critical strengths
The problem is accurately characterized, identifying the "waiting period" as a schedulable resource. The authors point out that the serial device-cloud handoff gives rise to two seemingly unrelated issues—inaccurate grounding after the response and idle computing power before it—which are fundamentally the same temporal scheduling problem. Addressing both phases surrounding the response in a unified manner offers a more structural approach than optimizing isolated points (such as upgrading the local model or reducing cloud latency) and explains why both latency and success rates can be improved simultaneously.
Critical weaknesses
-
Narrow experimental comparison. The only external baseline is EcoAgent, plus a fully-cloud upper bound. AAPT — the closest related work, which likewise precomputes during idle periods — is cited in a single sentence but never evaluated. The generality claim thus rests on "4 executors × 2 platforms" rather than on comparison against competing mechanisms, leaving the incremental value of two-sided preparation over existing idle-time precomputation unquantified.
-
Confounded headline result. Nearly all of the success gain comes from the post-response visual tools (+8.7 / +4.8 pt), while the pre-response prefetching the title emphasizes only cuts latency — yet Table 1 merges them into a single +5.7 pt. The cell that isolates the tools (tools-on / prefetch-off, i.e. "EcoAgent + tools") stays in the ablation rather than the main table.
-
Lack of fine-grained validation and failure analysis at the mechanism level. Seven tools employed only a coarse-grained "fully enabled/fully disabled" ablation approach, making it impossible to determine which specific components contributed to the results or which were rarely triggered.
Questions for authors
Is there any "false hit" — a prefetched action whose goal did not match the returned plan, yet still passed the semantic gate and the fresh-screen check and was executed? What is the false-hit rate, and what were the consequences?
Reponse To Reviewer EEr5
Thank you for your insightful review and for recognizing the value of treating cloud waiting time as a schedulable resource. Your recognition of our unified treatment of grounding and idle time reinforces the value of two-sided preparation for improving task success and reducing execution latency. We respond to your concerns below with additional comparisons, per-tool ablations, and an analysis of false hits.
A1: Thank you for raising this point. The key difference from AAPT is that TAP-Agent conditions prefetching on the actual post-action screenshot and retrieved experience, rather than preparing branches before the UI transition occurs.
We have compared TAP-Agent with AAPT using UI-TARS and OS-Atlas on AndroidWorld. Table 6 shows higher prefetch hit rates and lower execution time with both executors. These results support the advantage of preparing candidates from observed UI states and experience, allowing more prefetched work to be reused. We have added this comparison and expanded the distinction from AAPT in the revised manuscript (Section 5.4, Table 7).
Table 6: Comparison with AAPT on AndroidWorld using two executors. Time is the mean successful-episode duration.
A2: Thank you for highlighting the distinct contributions of these components. Due to the length constraints of the main text, we have added a comparison of EcoAgent + tools (tools on, prefetching off), EcoAgent, and TAP-Agent to the revised manuscript (Appendix D.1, Table 10), with success rates and execution times presented together. The two modules are also isolated in the main-text ablation in Section 5.3 (Table 4).
We have also changed the title to “TAP-Agent: Visual Tool Orchestration and Action Prefetching for Device–Cloud GUI Agents” to reflect both contributions: visual-tool orchestration improves task success, while action prefetching reduces interaction latency.
A3: Thank you for your valuable suggestion. We have measured tool usage and conducted individual tool-removal ablations with UI-TARS on both benchmarks. Table 7 reports calls during one full benchmark run and SR after removing each tool. Full TAP-Agent achieves 39.7% on AndroidWorld and 34.1% on OSWorld.
OmniParser is used most frequently on both platforms, and removing it leads to the largest drop in SR. Tool usage also varies by platform: zooming accounts for 21.7% of tool calls on OSWorld versus 13.8% on AndroidWorld, and removing it lowers OSWorld SR by 3.2 percentage points, showing its value in desktop environments. We have included the tool usage statistics, individual-tool ablations, and this analysis in the revised manuscript (Section 5.3, Table 6).
Table 7: Tool usage and individual removal ablations. SR in %.
A4: Thank you for your insightful question. We have audited the matcher's paired goal instructions using GPT-6.1 on AndroidWorld with UI-TARS. Among 850 prefetching rounds, 527 were accepted as hits (62.0%). GPT-6.1 identified 8 accepted hits as semantically mismatched, giving a semantic false-hit rate of 8 / 527 = 1.52%.
These false hits led to states in which the current plan could not continue. TAP-Agent handles such states through cloud replanning, at the cost of additional time and steps. We have added this semantic audit to Appendix D.3 of the revised manuscript.