acceptodds
Withdrawn from ICLR 2027

GUI-Drop: State Transition-Aware Visual Token Pruning for Multimodal GUI Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models enable GUI agents to interact with graphical interfaces by encoding high-resolution screenshots into long visual token sequences (e.g., 2,550 tokens for a mobile screenshot). However, most visual tokens are found redundant or irrelevant to action generation, incurring high computational and latency overhead. While existing training-free pruning methods reduce visual tokens, they mostly focus on intra-frame token relevance and redundancy within individual static screenshots. They neglect the logical continuity across GUI state transitions and the complementary cues of UI elements. In particular, they treat localized action feedback identically to static elements, while fragmenting UI elements into disjointed patches that slice complete text and decouple icons from their accompanying labels. To address these limitations, we propose GUI-Drop, a training-free token pruning framework that integrates transition-aware token selection with local feature compensation. We assign higher retention quotas to dynamic regions identified by inter-frame patch similarity and guide token selection by fusing current instruction-to-vision attention with spatially aligned attention signals from prior action generation. To preserve fine-grained local UI evidence, we balance token selection with attention-weighted feature diversity and compensate discarded representations into spatially adjacent retained neighbors. Across diverse online and offline GUI benchmarks and multiple VLM backbones, GUI-Drop generally outperforms existing training-free pruning methods. On AndroidWorld with Qwen3-VL-8B, GUI-Drop outperforms the unpruned baseline at 40% visual token retention. With UI-Voyager, our approach achieves a 59.5% success rate at 20% retention, compared with 43.1% for the strongest baseline, while reducing LLM prefill FLOPs by 41.7% at 40% retention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI agents, visual token pruning, vision-language models, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gU7DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GUI-Drop, a training-free method that prunes visual tokens for GUI agents. It keeps more tokens where the screen changed. At 20% of tokens, it clearly beats other pruning methods on AndroidWorld.

Critical strengths

(1) The gains over training-free baselines are large at low budgets. At 20% retention, UI-Voyager reaches 59.5% on AndroidWorld, versus 43.1% for the best baseline.

(2) The design fits GUI agents, and the cost report is honest. It uses screen changes and previous-action attention. It reports prefill, decode, and total time, not only FLOPs.

Critical weaknesses

(1) The end-to-end gain is small and costs accuracy. At 40% retention, per-step time only falls from 6.31 s to 5.78 s (1.09x), because prefill is about 11% of step time and decoding dominates. UI-Voyager also drops from 80.2% to 75.9% on AndroidWorld, and latency is measured only with eager attention.

(2) The state-transition component is weakly supported. Removing cross-step relevance or dynamic priority costs only 1.4 and 1.0 points, measured offline on AndroidControl, not on AndroidWorld. AndroidWorld results come from a single run over 116 tasks, so the 46.6% versus 43.1% gain over unpruned Qwen3-VL-8B is about four tasks.

Questions for authors

(1) Can you report AndroidWorld success over at least three runs with variance, and run the component ablation on AndroidWorld? A split by transition size would show when the transition signal helps.

(2) How does GUI-Drop compare with a lower-resolution screenshot of the same token count, and what is the per-step latency with FlashAttention 2? Were the 0.996 similarity threshold and the dynamic weight of 2.5 tuned on AndroidWorld tasks?

Reviewer_B6NGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GUI-Drop, a training-free token pruning method for GUI agents. Previous token reduction works focus on intra-frame signals and overlook inter-frame information, where this work fill this gap by integrating transition-aware token selection with local feature compensation. Experiments are conducted on various benchmarks including AndroidWorld and AndroidControl. It compares its method with many visual token pruning baselines and shows that GUI-Drop generally outperforms them, especially under aggressive compression rate. This work shows the potential of utilizing the inter-frame information to perform efficient GUI agent inference.

Critical strengths
  1. A GUI-specific use of inter-frame information. Most training-free pruning methods score tokens from a single frame, using attention, feature diversity, or spatial coverage. Prior GUI and video work that does use inter-frame information mainly treats it as redundancy to remove, for example by compressing history frames. GUI-Drop instead uses the change induced by the agent’s previous action as a saliency prior for the current frame, and it reuses historical decode attention only in pixel-stable regions. This state-transition-aware view of GUI token pruning has not been thoroughly studied. The modules are carefully designed and fully specified in the appendix, which makes GUI-Drop a solid baseline for follow-up work.

  2. Promising results in the closed-loop setting. GUI-Drop shows a consistent gain over existing pruning methods on AndroidWorld, especially at aggressive compression rates. With UI-Voyager it reaches 59.5% vs. 43.1% for the strongest baseline at 20% retention, and at 40% it leads all baselines on all five general backbones. The paper also profiles both LLM prefill and end-to-end latency (1.92× prefill speedup, 1.09× per step), which gives a transparent picture of the method’s practical cost and benefit. Offline benchmarks barely separate the methods, so these closed-loop gains are the main evidence supporting acceptance.

Critical weaknesses
  1. Limited analysis on methodology and robustness. The method proposed in this paper consists of several modules: budget allocation between dynamic and static regions, anchor-conditioned selection and local feature aggregation. Although an ablation study is conducted in Section 4.3, the overall structure remains heuristics. It has not made clear why the author design such structure and where the benefits really come from. The hyper-parameters, are also selected empirically without any robustness analysis. What's more, the ablation study is merely run on AndroidControl-High, which is offline and single-step. Some of the benefits the paper discussed, including error accumulation, would not happen, and the paper's main claim focus on online close-loop settings.
Questions for authors
  1. Can the authors extend the ablation study on AndroidWorld, including a single-frame variant with no partition and no historical attention? It would also be helpful to discuss the reason why each of the module is selected and designed.
  2. Can the authors elaborate how exactly are these hyper-parameters selected? Appendix A.2 states that a preliminary exploration is conducted, but as the large number of hyper-parameters, it remains unclear whether these parametes generalizes to different situations and whether the performance stays robust under different parameters.
  3. Why are VLA-Pruner and DivPrune absent from Figure 3 and Tables 2, 10, and 11?
Reviewer_sT7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GUI-Drop, a training-free visual token pruning framework for improving the inference efficiency of GUI agents. GUI-Drop is designed to mitigate two claimed limitations of existing pruning approaches: insufficient consideration of state transitions across adjacent frames and disruption of local contextual cues within UI elements. Specifically, GUI-Drop consists of three stages: budget allocation between dynamic and static regions with relevance estimated from current attention and historical decoding signals, anchor-conditioned selection that balances high-relevance tokens with diverse context carriers, and local feature aggregation that merges discarded neighboring representations into retained tokens. Experiments across online and offline GUI benchmarks, multiple VLM backbones, and different visual token retention ratios demonstrate performance improvements over existing pruning baselines. The authors further investigate component contributions and inference efficiency. Yet, the degradation introduced by pruning is concerning and can not be neglected.

Critical strengths
  1. The paper considers the distinctive characteristics of GUI interactions, including discrete state transitions and the complementary visual cues within GUI elements.
  2. The paper evaluates GUI-Drop across multiple GUI benchmarks, model backbones, and visual token retention ratios. The improvements over existing pruning baselines are encouraging. Additionally, the presentation of this paper, including tables and figures, is clear and provides good readability.
Critical weaknesses
  1. Limited targeted analysis of the identified GUI-specific limitations. The Introduction identifies two limitations of existing pruning methods: overlooking state transitions and fragmenting complementary GUI elements. Although these observations are intuitive and Figure 1 provides illustrative examples, their impact on downstream action prediction is not directly characterized. Prior GUI-agent studies, such as [1], [2], [3], and [4], demonstrate how preliminary experiments or failure-mode analyses can provide empirical support for problem formulation. While the component ablations in Table 3 support the effectiveness of the proposed method, additional targeted analyses of existing evaluation trajectories would help clarify how the proposed mechanisms address these two specific failure modes.
  2. Limited practical benefits under the reported efficiency-accuracy trade-off. Although GUI-Drop outperforms competing pruning methods, Table 1 in Section 4.2 shows substantial performance degradation compared to the unpruned baselines under aggressive compression. At 20% visual token retention, AndroidWorld success rates decrease from 60.3% to 26.7% for Step-GUI-4B and from 80.2% to 59.5% for UI-Voyager. Even at 40% retention, success rates decrease by 6.9% and 4.3%, respectively. Meanwhile, Table 4 in Section 4.4 reports only a 1.09x improvement in total model inference time, despite a 41.7% reduction in decoder prefill FLOPs. While these results do not negate the value of visual token pruning, they raise questions about the practical trade-off between computational savings and task performance, particularly for multi-step GUI interactions.

References

[1] MVP: Multiple View Prediction Improves GUI Grounding

[2] See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles

[3] Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

[4] Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

Questions for authors
  1. Regarding the identified limitations, could the authors provide further analysis of how the two failure modes illustrated in Figure 1 affect downstream action prediction? For example, representative failure cases or token retention statistics from existing evaluation trajectories could help clarify how GUI-Drop addresses overlooking state-transition and fragmenting complementary GUI elements.
  2. Regarding visual token redundancy, Table 1 shows substantial performance degradation when retaining only 10-20% of visual tokens, even with GUI-Drop. How do the authors reconcile these observations with the assumption that most visual tokens are redundant or irrelevant to GUI action generation? Could the authors further analyze whether the degradation results from insufficient token selection quality or from the loss of inherently necessary visual information?
  3. Regarding practical efficiency, given the modest 1.09x total inference speedup at 40% retention, which retention regime do the authors consider most practical for deployment? Could the authors provide available latency measurements at other retention ratios or further discuss the trade-off between inference efficiency and task success?
Reviewer_4rs7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GUI-Drop, a training-free token pruning framework that integrates transition-aware token selection with local feature compensation for multimodal GUI agents. It assigns higher retention quotas to dynamic regions identified by inter-frame patch similarity and combines current instruction-to-vision attention with attention signals from prior action generation. Attention-weighted feature diversity and spatial neighborhood aggregation are used to preserve fine-grained local UI evidence. The method is evaluated across online and offline mobile GUI benchmarks and multiple VLM backbones.

Critical strengths
  1. The paper addresses a practically relevant problem: reducing visual-token overhead in GUI agents.
Critical weaknesses
  1. Limited evaluation on high-resolution desktop interfaces. Visual token pruning is particularly valuable for high-resolution computer-use tasks, yet the evaluation mainly covers mobile benchmarks. Two representative benchmarks in CUA, Screenspot Pro and OSWorld should be evaluated.

  2. Limited evaluation beyond GUI tasks. Evaluating exclusively on GUI tasks may favor GUI-Drop’s domain-specific design over general-purpose baselines, raising concerns about comparison fairness. Including standard vision-language benchmarks would provide a more balanced assessment and clarify whether the reported gains generalize beyond GUI settings.

  3. Missing downsampling baseline. The paper does not establish whether GUI-Drop offers a better accuracy–efficiency trade-off than simply reducing screenshot resolution.

  4. Unclear novelty over VLA-Pruner. GUI-Drop shares VLA-Pruner core ingredients: current semantic attention, historical action attention, and diversity-based token selection. The authors should clarify their contributions.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.