acceptodds
Under review as a conference paper at ICLR 2027

DON’T LOOK TWICE: LEVERAGING VISUAL EXPERI- ENCE FOR TOKEN-EFFICIENT GUI AGENTS

Abstract

GUI agents powered by multimodal large language models (MLLMs) repeatedly process high-resolution screenshots throughout multi-step interactions, introducing substantial visual computation. Existing efficiency methods primarily reduce redun- dancy within individual screenshots or interaction trajectories, overlooking reusable visual experience accumulated across tasks. We introduce PAVE, a framework that leverages historical interactions for efficient GUI inference. PAVE retrieves relevant experience from previously completed tasks and uses it as a prior for a lightweight controller that adaptively selects visual tokens from the current screen, while keeping the underlying action model unchanged. Experiments on offline GUI benchmarks and online in-the-wild environments show that PAVE reduces visual-token computation by up to 17.4% and 8.3%, while retaining the original task performance. We open source our code and experiment to facilitate further research and application.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI Agent, Visual Token Pruning, Efficient Inference, Episodic Memory, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_fWgKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual-token reduction for GUI agents by reusing experience across completed episodes. PAVE maintains a persistent experience bank containing retrieval embeddings and, for eligible records, retained-token indices. Given a new screen and task, it retrieves similar records using CLIP and OCR-derived representations, applies a similarity threshold, and projects retrieved masks onto the current token grid to construct a spatial prior. A two-stage controller combines this prior with current-screen features and task information to select visual tokens, while the GUI action model remains frozen.

The controller is trained using verified sparse masks: within a predefined candidate family constructed from annotated target regions and additional screen cues, the method selects the smallest candidate that preserves a correct grounding prediction. Evaluation includes matched-budget grounding on GUIOdyssey and AndroidControl, interactive task success on AndroidWorld, a fixed-bank evaluation, transfer to a Qwen2.5-VL-7B backbone, retrieval ablations, and runtime measurements. The reported results show visual-token reductions with broadly similar observed accuracy and task success rates to the unmodified backbone, although the online confidence interval does not establish performance equivalence.

Critical strengths
  1. A useful focus on persistent experience for visual-token selection. The paper distinguishes cross-episode reuse from pruning within an individual screenshot and reusing computation within a trajectory. This is a useful framing of the contribution. The app-visit analysis is also informative: reported token reduction increases from 0.71% on the first visit to 13.82% on the eighth and subsequent visits (Section 5.2; Table 6, Appendix E.3). This pattern is consistent with increasing availability of reusable experience, although it does not by itself establish that token selection improves as the bank grows.

  2. A concrete supervision procedure and several useful evaluation controls. The verified-mask construction (Eq. 7) derives token-selection targets from existing bounding-box annotations and the frozen model's responses, without requiring additional manual token-level annotations. The evaluation fixes several important components, including the backbone, prompt, parser, and controller checkpoint, and reports paired online comparisons with uncertainty estimates. The fixed-bank evaluation helps assess performance when the initial experience bank is held fixed. Random-History Retrieval is particularly useful: accuracy falls from 43.35% to 40.90% on GUIOdyssey and from 66.86% to 62.13% on AndroidControl. These results support using relevant rather than randomly selected history, while leaving open the incremental benefit over a controller that receives no historical prior.

Critical weaknesses
  1. The incremental contribution of retrieved experience remains insufficiently isolated from the controller's learned token-selection ability. The supervision in Eqs. 7 and 12 is derived from candidate masks containing an annotated target region. This gives the controller a direct localization-related learning signal. It does not imply that the controller ignores history, but it makes a comparison with a controller-only baseline important. Reporting how often each candidate supplies the target mask would also help characterize the supervision; the construction alone does not establish which candidate is typically selected.

Random-History Retrieval provides relevant evidence, but it compares useful history with potentially misleading history. It does not establish the gain over having no historical prior. The page-template states in Appendix A.2 offer a useful descriptive analysis: the controller can run with a zero spatial prior, and Table 5 reports 1,576 such steps at 2,197.2 tokens per decision, compared with 2,550.0 for the no-memory category and 1,913.4 for mask-bearing records. Accuracy on these strata would be informative, but differences in their screen and task distributions prevent this comparison from isolating the effect of the prior.

A stronger test would compare relevant, zero, and shuffled priors on the same held-out states, using the same controller and matched retained-token budgets. This would measure the existing controller's dependence on the prior. A separately trained controller without historical inputs, using the same training data and comparable capacity, would address whether similar accuracy–efficiency trade-offs can be achieved without experience retrieval. Distinguishing these two questions would substantially strengthen the central attribution claim.

  1. Robustness to stale experience is not established. PAVE uses retrieved masks as inputs to a controller that also observes the current screen; it does not simply apply an old mask unchanged. Nevertheless, a previously useful spatial prior could become misleading after a layout, theme, or application-version change. The current evaluation does not establish how reliably the retrieval gate rejects such records, or how effectively the controller compensates when stale records are retrieved.

AndroidWorld's application selection and execution protocol impose constraints for reproducible evaluation, including local data storage, no required authentication, and fixed application versions (AndroidWorld, Section 3.3 and Appendix C.1). This limits what the benchmark establishes about deployment beyond those conditions. In particular, a fixed-version evaluation does not test whether persistent experience remains useful across application updates. That omission is directly relevant to PAVE's persistence claim. A controlled perturbation study, together with a description of any record invalidation or refresh policy, would address this concern. Describing these experiments as “in-the-wild” would overstate the deployment conditions examined.

Additional concerns, secondary to the two points above:

(a) The threshold sweep would benefit from a decomposition of its effects. Changing the similarity threshold can affect both the fraction of states that receive historical conditioning and the composition of the retrieved records. The reported accuracy ranges are 42.28–43.47% on GUIOdyssey and 66.81–66.91% on AndroidControl, corresponding to ranges of 1.19 and 0.10 percentage points, respectively. The cited token-reduction range of 2.27–17.40% is substantial, but these aggregate results do not identify the contribution of retrieval coverage versus selection quality. Please report, for each threshold, the conditioning rate, the number of mask-bearing records used, reduction on conditioned states, and uncertainty in accuracy. Without these quantities, the sweep supports an operating trade-off more directly than it explains retrieval relevance.

(b) Baseline coverage and operating-point comparisons could be stronger. Appendix B describes adaptations of FastV and VisionZip to GUI-Owl. Such adaptations may be necessary, but their departures from the original methods and any implementation checks should be made clear. Matching baselines to PAVE's per-state token budget is a useful control; additional operating points would show whether the comparison holds across accuracy–retention trade-offs. FocusUI is a relevant comparison for efficient UI grounding, subject to differences in training, backbone, resolution, and token accounting. The paper should also clarify its relationship to intra-trajectory efficiency methods and to cross-episode memory methods such as MementoGUI. These methods need not all be direct experimental baselines if their objectives or interfaces differ substantially, but those differences should be explained.

(c) Performance-preservation claims should reflect the uncertainty and evaluation scope. On AndroidWorld, the paired result is 123/255 successes for PAVE versus 125/255 for the backbone, a difference of approximately −0.8 percentage points, with a reported 95% confidence interval of [−3.8, 2.5] and McNemar p = 0.804. This is compatible with similar performance, but also with a practically relevant loss; failure to detect a difference does not establish equivalence. The fixed-bank result, 33/71 versus 34/71, should be interpreted with the same caution. The abstract would be more precise if it described similar observed task success rather than implying established performance equivalence.

The offline evaluation focuses on click grounding, whereas AndroidWorld supplies end-to-end task evidence. The limitation is therefore the breadth and precision of that evidence, rather than a complete absence of task-level validation. A breakdown by action type and task category would help characterize where performance is preserved. Appendix F's runtime comparison is useful, but a breakdown of total per-decision latency and peak memory, including auxiliary components, would make the practical benefit of token reduction easier to assess.

(d) The scope of transfer and the offline–online gap need clearer characterization. App-scoped retrieval can reuse experience across different tasks within the same app; it does not, by itself, demonstrate cross-app transfer. Please specify whether “across tasks” refers to distinct task instances, task templates, or broader transfer. Similarly, the reduction from 14.93%/17.40% offline to 8.29% online could reflect differences in bank coverage, screen distributions, or other protocol factors. More diverse interfaces are a plausible explanation, but a decomposition by coverage and conditioned-state reduction would provide stronger evidence.

(e) Bank maintenance and storage implications need more detail. Appendix E.2 specifies a fixed bank of 477 records, including 351 page templates and 126 sparse masks. Please also report how bank size and retrieval latency evolve during the online runs, and describe any capacity limit, deduplication, eviction, or refresh policy. Storing embeddings instead of raw screenshots changes the retained information but does not by itself establish privacy protection. A brief account of what is stored, where it is stored, and how it can be deleted would help assess the deployment implications.

Questions for authors
  1. Contribution of the historical prior. Can you compare relevant, zero, and shuffled priors on the same states with matched retained-token budgets? Please distinguish this inference-time ablation from a controller trained without historical inputs. Accuracy for the zero-spatial-prior page-template stratum in Table 5 would also be useful, but would not alone isolate the prior's contribution. Which candidate in Eq. 12 most often supplies the supervision target?

  2. Retrieval coverage. At each threshold, what fraction of decisions is conditioned, how many mask-bearing records are used, and how much pruning occurs on conditioned decisions? Can you report uncertainty in the accuracy differences and distinguish changes in retrieval coverage from changes in selection quality?

  3. Comparable efficiency baselines. Can you include a comparison with FocusUI under a compatible setup, or explain the relevant incompatibilities? FocusUI reports a decrease from 48.3% to 45.1% on ScreenSpot-Pro at its nominal 30% retention setting, together with runtime and memory measurements. These results are not directly comparable to PAVE's current benchmarks. A comparison should align the task, backbone, training assumptions, image resolution, and token-accounting conventions as far as possible, and report several operating points.

  4. Stale experience. Can you evaluate a bank built before a controlled interface change? On the same perturbed screens, compare a zero prior, an old prior, and an updated prior, with matched token budgets where feasible. Report the normal retrieval pipeline separately, including its conditioning rate, since gate rejection is itself a potential robustness mechanism. Forced retrieval of old records would be a useful additional stress test, but should be distinguished from normal deployment behavior. The experiment should also measure the current-screen controller's sensitivity to the perturbation itself.

  5. Full-pipeline cost. Does the runtime accounting in Appendix F include all auxiliary stages? Please report total per-decision latency, peak memory, and a component breakdown for the visual encoders, OCR where used, retrieval, the controller, and action generation. Throughput or FLOPs would be useful where they match the intended deployment setting, but clear end-to-end latency accounting is the first priority.

Suggestions for future work, rather than requirements for this assessment:

(a) Combine cross-episode token selection with compatible intra-trajectory computation reuse and evaluate the joint accuracy–efficiency trade-off.

(b) Report an annotation-assisted reference within a clearly specified candidate family. This could help contextualize achievable savings under that restricted search, without presenting it as the globally smallest sufficient token subset.

7 comments

We thank the reviewer for the constructive feedback and insightful questions. Below, we provide detailed responses to address these concerns, and we hope our additional experimental results and clarifications help resolve these concerns. We will also incorporate the new results and corresponding discussions into the revised manuscript. Thank you very much.

Q1: Contribution of the historical prior. Can you compare relevant, zero, and shuffled priors on the same states with matched retained-token budgets? Please distinguish this inference-time ablation from a controller trained without historical inputs. Accuracy for the zero-spatial-prior page-template stratum in Table 5 would also be useful, but would not alone isolate the prior's contribution. Which candidate in Eq. 12 most often supplies the supervision target?

We appreciate the reviewer's suggestion to distinguish the contribution of historical priors. To isolate the effect of historical priors, we propose an additional controlled ablation using the same held-out states, frozen backbone, controller checkpoint, and per-state visual-token budgets. Specifically, we compare three inference-time conditions: (1) PAVE (relevant); (2) zero; and (3) shuffled. We additionally consider a separately trained no-history controller, which receives only the current visual features, token positions, and task embedding, using the same supervision data and comparable model capacity.

ConfigurationGUIOdysseyAndroidControl
PAVE52.40%52.36%
Zero43.73%50.55%
Shuffled50.06%50.55%
Trained no-history controller47.60%49.45%

As shown in the results, PAVE consistently achieves the best performance across both benchmarks. Compared with the zero-prior setting, incorporating relevant historical spatial priors improves accuracy by 8.68% on GUIOdyssey and 1.82% on AndroidControl, demonstrating the additional benefit of historical experience beyond the controller's current-screen token-selection capability. In contrast, replacing relevant priors with shuffled priors leads to accuracy drops of 2.34% and 1.82%, respectively, highlighting the importance of retrieving relevant rather than arbitrary historical information. Furthermore, PAVE outperforms the separately trained no-history controller by 4.81% and 2.91%, suggesting that incorporating historical experience provides additional benefits beyond those achievable through controller training alone. Regarding the supervision construction in Eq. 12, the candidate family combines the target region with optional OCR, layout, masks, etc. Among the 4,212 selected verified sparse targets, 18.92% are derived from candidates containing the target region with the top and bottom layout bands (Gt∪UtG_t \cup U_t), 18.87% from candidates containing the target region alone (GtG_t), and 25.83% from candidates incorporating projected masks from eligible training records.

Q2: Retrieval coverage. At each threshold, what fraction of decisions is conditioned, how many mask-bearing records are used, and how much pruning occurs on conditioned decisions? Can you report uncertainty in the accuracy differences and distinguish changes in retrieval coverage from changes in selection quality?

We appreciate the suggestion to distinguish retrieval coverage from token-selection quality. The retrieval threshold influences both the proportion of decisions receiving historical conditioning and the composition of retrieved experiences. To examine these effects, we report the controller-conditioning rate, number of mask-bearing retrievals, and token reduction on conditioned decisions at each threshold.

DatasetThresholdConditioned decisionsMask-bearing retrievalsReduction on conditioned decisions
GUIOdyssey0.7593.53%1,70519.20%
GUIOdyssey0.8084.25%1,55319.53%
GUIOdyssey0.8569.64%1,33921.00%
GUIOdyssey0.9048.50%1,00223.96%
GUIOdyssey0.9524.37%54526.14%
AndroidControl0.7572.49%2,18023.86%
AndroidControl0.8054.28%1,84426.26%
AndroidControl0.8536.22%1,31428.47%
AndroidControl0.9019.60%79029.61%
AndroidControl0.957.22%32330.53%

As the threshold decreases from 0.95 to 0.75, the conditioning rate increases substantially from 24.37% to 93.53% on GUIOdyssey and from 7.22% to 72.49% on AndroidControl, while token reduction on conditioned decisions remains relatively stable (19.20%–26.14% and 23.86%–30.53%, respectively). These results suggest that the greater overall token savings at lower thresholds are primarily driven by broader retrieval coverage rather than more aggressive token pruning on individual decisions.

To assess the reliability of the accuracy differences, we conduct paired accuracy comparisons using state-level bootstrap resampling. The resulting 95% confidence intervals for accuracy differences relative to the selected operating points include zero across all evaluated thresholds, indicating that no statistically significant accuracy differences are detected.

Q3: Comparable efficiency baselines. Can you include a comparison with FocusUI under a compatible setup, or explain the relevant incompatibilities? FocusUI reports a decrease from 48.3% to 45.1% on ScreenSpot-Pro at its nominal 30% retention setting, together with runtime and memory measurements. These results are not directly comparable to PAVE's current benchmarks. A comparison should align the task, backbone, training assumptions, image resolution, and token-accounting conventions as far as possible, and report several operating points.

We thank the reviewer for suggesting FocusUI as an additional baseline. To provide a more comprehensive comparison, we conduct additional experiments comparing PAVE with FocusUI under two operating conditions: (1) comparable visual-token reduction rates and (2) comparable click accuracies.

At similar visual-token reduction rates (14.93% for PAVE vs. 14.55% for FocusUI), PAVE and FocusUI achieve click accuracies of 43.35% and 33.29%, respectively, representing a 30.2% relative improvement over FocusUI. Furthermore, at comparable click accuracies (32.69% for PAVE vs. 32.87% for FocusUI), PAVE achieves a substantially higher visual-token reduction rate (49.76% vs. 24.07%), corresponding to a 106.7% relative improvement. These results demonstrate that PAVE achieves a more favorable accuracy–efficiency trade-off.

We also thank the reviewer for highlighting relevant work. Intra-trajectory efficiency methods primarily exploit temporal redundancy within individual episodes, whereas PAVE leverages persistent cross-episode experience for token selection. Unlike cross-episode memory methods such as MementoGUI, which focus on improving action prediction, PAVE uses historical experience to reduce visual-token computation without modifying the action model. We will further clarify these distinctions in the revised manuscript.

Reviewer_K7RkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAVE retrieves experience from completed GUI tasks and uses stored token masks as spatial priors for a trained token-selection controller. Its training masks are verified by rerunning a frozen action model. Selected post-encoder tokens retain their original positions, and the action model remains unchanged.

Critical strengths
  1. PAVE stores compact masks of useful screen regions instead of adding old screenshots to the action model's context. The controller still checks the current screen before choosing tokens. This is a practical way to reuse familiar interface layouts without changing the action model (Section 3.4; Appendix A.3).

  2. Training masks are checked by running the action model, not just drawn around the target button. This can preserve nearby text and navigation elements needed to choose the right action. PAVE saves tokens while keeping similar grounding accuracy on both offline benchmarks (Appendix A.4; Table 1).

Critical weaknesses
  1. Historical masks are transferred by resizing their coordinates, not by matching interface elements (Section 3.4; Appendix A.2). This works most naturally when the layout stays similar. After scrolling, a list reorder, or a popup, an old mask may point to the wrong content even if the screen still has a high retrieval score. The current-screen controller may correct this, but the paper does not show how reliably it does so when relevant elements move. This matters for reusing experience across changing interfaces.

  2. Useful mask experience is difficult to build from an empty bank. A mask is stored only after a successful episode and only if the controller already kept at most 80% of the tokens (Appendix A.2). Full-token decisions add page templates, not masks. In the separate main-run telemetry, only 13.4% of calls have retrieved sparse masks available, while 37.2% have page templates only (Table 5). This limits how often the proposed spatial memory can help. The visit-count results suggest improvement with use, but do not show whether mask coverage keeps growing across different pages or remains concentrated on a few familiar layouts.

Questions for authors
  1. When a relevant button moves after scrolling or a layout change, does the controller discard the old spatial prior and retain the new location? Please report pruning errors on similar-looking screens with moved targets.

  2. As the bank grows, how many distinct pages gain usable masks? How often does the controller turn a page-template-only retrieval into a new stored mask, and do those masks help on later tasks?

2 comments

We thank the reviewer for the constructive feedback and insightful questions. Below, we provide detailed responses to address these concerns, and we hope our additional experimental results and clarifications help resolve these concerns. We will also incorporate the new results and corresponding discussions into the revised manuscript. Thank you very much.

Q1: When a relevant button moves after scrolling or a layout change, does the controller discard the old spatial prior and retain the new location? Please report pruning errors on similar-looking screens with moved targets.

We thank the reviewer for this insightful suggestion. To examine whether PAVE remains effective when interface elements change positions, we conduct an additional evaluation on similar-looking screens with moved targets within the offline benchmarks. Specifically, two authors independently annotate screenshots containing repositioned buttons and resolve disagreements through discussion. This process yields 100 screenshots with moved targets, including 89 from GUIOdyssey and 11 from AndroidControl. Our results show that PAVE correctly predicts the target in 95.0% cases, including 95.5% on GUIOdyssey and 90.9% on AndroidControl. These results suggest that PAVE can preserve accurate target grounding despite changes in element positions. This robustness stems from our controller design: historical masks serve only as soft spatial priors, while the final token-selection decisions are jointly determined by current-screen visual features, spatial positions, and task embeddings.

We further conduct a controlled perturbation experiment in the online AndroidWorld environment. Specifically, we randomly scroll the interface by 10%–50% of the screen height, obtaining 85 perturbed observations. Among these, PAVE successfully activates the controller and correctly predicts the target in 89.4% cases. In the remaining 9 cases, substantial changes in screen content cause the retrieval similarity to fall below the threshold, preventing PAVE from activating sparse token selection. These cases therefore trigger conservative full-token processing rather than applying potentially outdated spatial priors.

Together, these findings indicate that PAVE can accommodate moderate target displacement through current-screen-conditioned token selection, while its similarity-based fallback helps avoid unreliable historical reuse under larger interface changes. We will incorporate these results and representative examples into the revised manuscript.

Q2: As the bank grows, how many distinct pages gain usable masks? How often does the controller turn a page-template-only retrieval into a new stored mask, and do those masks help on later tasks?

We appreciate the reviewer's concern regarding experience-bank growth. We first clarify that page-template-only retrieval does not prevent the controller from generating new sparse masks. As described in Appendix A.2, page templates provide retrieval evidence even without spatial masks, allowing the controller to perform token selection based on the current screen and task. Once the episode succeeds, controller decisions retaining no more than 80% of visual tokens can be stored as new sparse-mask records, making them available for subsequent interactions.

To further investigate experience accumulation, we analyze the coverage and reuse of sparse masks across distinct page families. Across an average of 179.3 distinct page families in the final experience banks of three independent AndroidWorld evaluations, 28.3% acquire at least one usable sparse mask.

The table below reports the growth of mask-covered page families for each of the three independent runs, together with their mean. We use the same Round 1–3 labels as in Table 4.

Page families with usable sparse masks

Run25%50%75%100%
Round 19203545
Round 210253850
Round 39214057
Mean9.322.037.750.7

Percentages indicate progress through successful episodes that added records to the bank, in chronological order.

Furthermore, among 538 steps involving page-template-only retrieval and controller invocation, 249 (46.3%) produce new sparse-mask records, of which 114 (45.8%) are subsequently retrieved in later interactions. These observations suggest that mask experience progressively expands across interface pages rather than remaining confined to a small number of familiar layouts.

This finding is also consistent with our experimental observations in Table 5. In the main-run logs, 13.4% of decisions have retrieved sparse masks available, whereas this proportion reaches 29.8% in the additional fixed-bank evaluation. The final fixed bank contains 477 records, including 126 sparse-mask records. Although the evaluation settings differ, these results illustrate the benefit of accumulated experience and the ability of PAVE to leverage reusable visual information over continued interactions. As future work, we plan to conduct a longitudinal deployment study to systematically investigate the long-term growth, coverage, and reuse of visual experience across diverse applications and evolving interfaces.

Comments on the paper

6 comments

We thank the reviewer for the constructive feedback and insightful questions. Below, we provide the detailed responses to address these concerns, and we hope our additional experimental results and clarifications help resolve these concerns. We will also incorporate the new results and corresponding discussions into the revised manuscript. Thank you very much.

Q1.Token reduction vs. actual latency: The runtime comparison shows only a 0.5% improvement (3.743s vs 3.762s), despite 8-17% token reduction. Could you elaborate on why the token savings do not translate to more significant latency improvements? Is this due to the overhead of the controller and retrieval modules, or are the visual tokens not the bottleneck in the overall inference pipeline? Understanding this would help assess the practical value of the approach.

We thank the reviewer for raising this important question. To further investigate the discrepancy between visual-token reduction and end-to-end latency improvements, we conduct a detailed component-level analysis on an NVIDIA RTX A6000. Specifically, PAVE reduces median prefill latency by 28.2% (1.39× speedup) when the controller is activated, demonstrating the computational benefit of visual-token pruning. The overall improvement is moderated by other components, including OCR (0.578 s), retrieval embedding computation and similarity search (0.0715 s), spatial-prior construction and controller inference (0.0058 s), visual encoding (0.329 s), and action generation (0.442 s for prefill and 1.648 s for decoding). We expect that further optimizations in hardware acceleration could help reduce these overheads and improve end-to-end efficiency, which we will explore in future work.

Q2. Controller behavior on out-of-distribution screens: The paper evaluates on held-out test states from the same benchmarks, but does not analyze what happens when the controller encounters screens that are very different from anything in the experience bank. Could the controller make catastrophically bad pruning decisions in such cases? Is there a fallback mechanism beyond the similarity threshold τ?

We thank the reviewer for this insightful question. PAVE employs a similarity-based retrieval gate to prevent unreliable historical conditioning. Specifically, when no retrieved experience satisfies the similarity threshold, PAVE bypasses token pruning and falls back to full-token processing using the original frozen backbone.

To further examine robustness to interface changes, we conduct an additional evaluation on 100 similar-looking screenshots with moved targets, including 89 from GUIOdyssey and 11 from AndroidControl. PAVE correctly predicts the target in 95.0% cases. We further introduce random scrolling offsets of 10%–50% of screen height on 85 AndroidWorld in-the-wild observations. PAVE correctly predicts the target in 89.4% cases, while the remaining nine cases trigger full-token fallback due to insufficient retrieval similarity. These findings suggest that PAVE can accommodate moderate interface changes while conservatively avoiding historical conditioning when substantial changes are detected. Note that, historical masks act as soft spatial priors, while the controller makes token-selection decisions based on current-screen visual features and task information. We acknowledge that these perturbation experiments do not fully characterize the risk of incorrect pruning. We will clarify this limitation and explore additional confidence-aware fallback mechanisms in future work.

Q3. Comparison with stronger baselines: The baselines include Uniform Grid, Random Selection, Random History, FastV, and VisionZip. Have you compared against more recent GUI-specific methods like AQuaUI or ReVision? These methods are designed specifically for GUI screenshots and might provide a stronger baseline for intra-episode efficiency, making the cross-episode advantage of PAVE clearer.

We thank the reviewer for suggesting AQuaUI and ReVision. We conduct additional comparisons with these recent methods on AndroidWorld. The results are summarized below.

MethodEpisodeSuccess RateToken Reduction
PAVE1st45.36%8.90%
PAVE2nd47.42%20.31%
AQuaUI1st/2nd38.14%19.56%
ReVision-RTS1st/2nd35.14%−74.72%

PAVE outperforms both AQuaUI and ReVision-RTS in task success and visual-token efficiency in the cross-episode evaluation. Specifically, PAVE achieves 47.42% task success with 20.31% token reduction, outperforming AQuaUI by 9.28% in task success while achieving slightly greater token reduction (20.31% vs. 19.56%). Regarding ReVision-RTS, although it prunes tokens from the current screenshot, it additionally processes visual tokens from the previous screenshots, resulting in 74.72% higher token usage in total. In contrast, PAVE uses 54.39% fewer visual tokens than ReVision-RTS in the second episode. These findings demonstrate PAVE's potential for progressively improving efficiency through persistent cross-episode experience reuse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.