acceptodds
Withdrawn from ICLR 2027

Beyond Decoding Acceleration: Fast Plan-Conditioned Grounding for GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal GUI agents can automate web, desktop, and mobile interfaces from natural language instructions and screenshots, but their practical use is limited by high per-step latency. While existing acceleration methods mainly reduce subsequent-token decoding cost, GUI grounding typically produces only short actions and coordinates. Our latency analysis shows that the dominant bottleneck is instead the first-token latency caused by repeated visual encoding, multimodal fusion, and prefilling in large vision-language models. To address this, we propose PACE, a Plan-conditioned Action and Coordinate Executor, within a two-speed GUI agent framework. A main model generates concise next-step plans, and PACE executes simple and safe steps by directly predicting the action type and target location from the plan and screenshot. PACE is a lightweight 0.2B dual-stream model with cross-modal fusion and a grid-based grounding head. Across five GUI grounding benchmarks, PACE achieves 64.0% average accuracy with 46.59 ms latency. On OSWorld, integrating PACE reduces main-model calls and total execution time while largely preserving task performance. These results show that low-latency GUI agents require not only faster decoding, but also plan-conditioned delegation that avoids unnecessary heavy multimodal computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fast GUI Agent, Fast GUI grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EG9gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PACE, a lightweight executor that grounds next-step plans from a large VLM into GUI actions. It reports low grounding latency and reduced main-model calls and execution time on OSWorld.

Critical strengths
  • The paper addresses a practical bottleneck: high first-token latency in GUI agents.

  • PACE achieves a promising speed–accuracy trade-off, with evaluations covering both standalone grounding and end-to-end task execution.

Critical weaknesses
  • The acceleration mechanism is confusing. Sections 3.1 and 3.4 describe invoking the main VLM at every step before delegation, which would retain its prefill cost. Appendix D.1 instead suggests executing a previously generated plan on the next screenshot without another VLM call. This inconsistency obscures the central mechanism and the reported reduction in model calls.

  • Plan reliability is insufficiently addressed. Under the latter interpretation, the planner predicts the next action before observing the outcome of the current action. Unexpected interface changes can invalidate this plan. Although Appendix D.2 analyzes errors, the paper does not clearly describe how stale plans are detected and rejected before execution. The planner’s own delegation flag provides limited assurance.

  • The contribution appears incremental. The method reduces the frequency of expensive inference rather than improving prefill itself. Its advantage over inference optimizations, such as prefix/KV-cache reuse, remains unclear. If the main idea is to generate an additional plan during an existing call, the paper should also explain why it predicts only one subsequent step and compare against generating several future plans.

Questions for authors
  • Can you provide pseudocode showing exactly which environment steps invoke the main VLM? How should Equations (1)–(3) be reconciled with Appendix D.1?

  • Why generate only one future plan? How does this compare with producing multiple plans per VLM call?

  • How does PACE detect that a plan is no longer valid after the interface changes? What triggers recovery, and what is its latency cost?

  • Is prefix caching enabled in the baselines? Can you separate visual encoding, prefill, and decoding costs to establish the remaining bottleneck?

  • Why does Figure 1 report 16.28 s per step for the 32B model, whereas Table 2 reports 7.29 s for its baseline? Do the models, settings, or timing definitions differ?

  • Appendix E.3 cites ScreenSpot to a 2008 smart-space paper and Android in the Wild to a 2016 text-watermarking paper. Can these be corrected to the relevant ScreenSpot and Android in the Wild sources?

Reviewer_HgEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that per-step latency of VLM-based GUI agents is dominated by first-token latency rather than decoding. It proposes PACE, a two-speed framework where the main VLM planner outputs a next-step plan and a delegation flag, and simple steps are delegated to a 0.2B plan-conditioned executor with cross-attention fusion and a cell-plus-offset grounding head. PACE achieves 64.0% average accuracy at 46.59 ms on five grounding benchmarks. On OSWorld, it reduces main-model calls and total execution time across four Qwen3-VL planners.

Critical strengths
  • The latency decomposition in Figure 1 shows that TTFT accounts for most of the per-step latency. This is a useful observation that motivates acceleration beyond decoding.
  • The executor is simple and practical. It achieves a favorable speed-accuracy trade-off in the low-latency regime, and the ablations on fusion, grounding head, and encoder fine-tuning are clean.
  • The error attribution in Table 7 helps explain how planner quality affects delegation.
Critical weaknesses
  • Despite the title, there is no acceleration baseline, neither decoding-side (e.g., speculative decoding) nor prefill-side (e.g., visual token pruning such as FastV applied to the planner), even though Appendix E.2 claims efficient VLMs are complementary to PACE. It is also unclear whether the gains come from delegation or from PACE itself, since PACE is never replaced by compact grounders already in Table 1 (e.g., GoClick-L, UGround-V1-2B). And although PACE is trained on GUI-Actor data, GUI-Actor (version without verifier) models are absent from Table 1.
  • The OSWorld differences are small, often 1–2 points, which means verfy few cases, and come from a single run. Without mean and variance over multiple runs, it is hard to tell whether the score changes are meaningful.
  • Some large models in Table 1, e.g., Qwen3.5-plus, appear to be accessed via API. If so, network latency is mixed into the comparison and into Figure 2, and the deployment setting of each baseline should be described.
  • The delegation ratio and the accuracy of delegated steps are not reported, though they are the most direct indicators of how much execution is actually offloaded to PACE.
Questions for authors
  • How does PACE compare with, or combine with, a prefill-side acceleration method on the planner? How does the framework perform when PACE is replaced by GoClick-L or UGround-V1-2B? How does GUI-Actor perform in Table 1?
  • Can you report the mean and standard deviation over at least 3 runs for at least one planner?
  • How is each baseline in Table 1 deployed (local vLLM or API), and is network latency included in the reported times?
  • For each planner, what are the delegation ratio and the accuracy of delegated steps?
Reviewer_KR7VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method, called PACE with a lightweight 0.2B model designed to accelerate GUI agents by reducing expensive VLM calls. The key observation is that GUI agent latency is dominated by TTFT, rather than autoregressive decoding, making traditional decoding acceleration less effective. To address this, the authors introduce a two-speed framework in which a large VLM handles high-level reasoning and generates next-step plans, while PACE executes simple, low-risk actions by directly predicting action types and screen coordinates from screenshots and plans. PACE uses dual-stream visual and text encoders, cross-modal attention, and a grid-based grounding head for efficient action prediction. Experiments show that PACE achieves 64.0% average grounding accuracy with only 46.59 ms latency, reducing end-to-end execution time on OSWorld by approximately 14–22% while largely preserving task performance.

Critical strengths
  1. The paper identifies a key bottleneck in GUI agents: TTFT accounts for over 60% of per-step latency, making conventional decoding acceleration less effective. This observation motivates a shift from optimizing token generation to reducing unnecessary large VLM invocations. The clear empirical motivation and practical relevance strengthen the significance of the proposed approach.

  2. PACE achieves a speed-accuracy trade-off, reaching 64.0% average grounding accuracy with only 46.59 ms latency using a 0.2B model. More importantly, integrating PACE into different VLM planners reduces end-to-end execution time on OSWorld by approximately 14–22%, while largely preserving task performance.

Critical weaknesses
  1. Limited novelty in both the overall framework and model architecture. The planner-grounder decomposition is already a well-established paradigm in GUI agents, and the proposed two-speed framework does not introduce a fundamentally new approach to this architecture. The PACE model itself largely combines conventional components, including a ViT-based image encoder, a BERT-style text encoder, and cross-attention for multimodal fusion, followed by standard action classification and coordinate prediction heads. The delegation mechanism is also based on predefined rules or prompting rather than a novel learning-based approach.

  2. Insufficient experimental competitiveness and weak end-to-end performance. Although the paper reports favorable speed-accuracy trade-offs, the grounding evaluation does not sufficiently compare against the latest and strongest GUI grounding methods, making it difficult to establish the competitiveness of PACE against current state-of-the-art approaches. More importantly, the end-to-end OSWorld results remain relatively weak, with task scores ranging from approximately 17% to 29% across the evaluated configurations. While the proposed framework reduces latency, it does not demonstrate improvements in task success, and some configurations even exhibit performance degradation. These limitations weaken the practical significance of the reported efficiency gains, particularly for deployment in more capable GUI agents.

Questions for authors
  1. The separation of high-level planning and low-level grounding has been widely explored in prior GUI agent systems. Could the authors clarify what the fundamental methodological contribution of PACE is beyond this paradigm? In particular, given that the grounder primarily combines standard ViT/BERT encoders, cross-attention, and coordinate prediction heads, what distinguishes the proposed approach from existing lightweight GUI grounding architectures?

  2. The experimental evaluation does not appear to sufficiently cover the latest competitive GUI grounding approaches. Could the authors provide comparisons with stronger recent baselines under consistent input conditions, hardware, and latency measurement protocols? This would help establish whether PACE offers a genuinely superior speed-accuracy trade-off rather than benefiting from comparisons with relatively weak or outdated methods.

  3. The reported OSWorld task scores remain relatively low (approximately 17–29%), raising concerns about whether the proposed acceleration framework generalizes to more capable GUI agents. Could the authors evaluate PACE with stronger planners that achieve substantially higher baseline success rates on OSWorld and report the corresponding changes in task success, latency, and delegation errors? Such results would be important for assessing whether the efficiency gains remain meaningful in competitive real-world agent settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.