- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies visual-token reduction for GUI agents by reusing experience across completed episodes. PAVE maintains a persistent experience bank containing retrieval embeddings and, for eligible records, retained-token indices. Given a new screen and task, it retrieves similar records using CLIP and OCR-derived representations, applies a similarity threshold, and projects retrieved masks onto the current token grid to construct a spatial prior. A two-stage controller combines this prior with current-screen features and task information to select visual tokens, while the GUI action model remains frozen.
The controller is trained using verified sparse masks: within a predefined candidate family constructed from annotated target regions and additional screen cues, the method selects the smallest candidate that preserves a correct grounding prediction. Evaluation includes matched-budget grounding on GUIOdyssey and AndroidControl, interactive task success on AndroidWorld, a fixed-bank evaluation, transfer to a Qwen2.5-VL-7B backbone, retrieval ablations, and runtime measurements. The reported results show visual-token reductions with broadly similar observed accuracy and task success rates to the unmodified backbone, although the online confidence interval does not establish performance equivalence.
Critical strengths
-
A useful focus on persistent experience for visual-token selection. The paper distinguishes cross-episode reuse from pruning within an individual screenshot and reusing computation within a trajectory. This is a useful framing of the contribution. The app-visit analysis is also informative: reported token reduction increases from 0.71% on the first visit to 13.82% on the eighth and subsequent visits (Section 5.2; Table 6, Appendix E.3). This pattern is consistent with increasing availability of reusable experience, although it does not by itself establish that token selection improves as the bank grows.
-
A concrete supervision procedure and several useful evaluation controls. The verified-mask construction (Eq. 7) derives token-selection targets from existing bounding-box annotations and the frozen model's responses, without requiring additional manual token-level annotations. The evaluation fixes several important components, including the backbone, prompt, parser, and controller checkpoint, and reports paired online comparisons with uncertainty estimates. The fixed-bank evaluation helps assess performance when the initial experience bank is held fixed. Random-History Retrieval is particularly useful: accuracy falls from 43.35% to 40.90% on GUIOdyssey and from 66.86% to 62.13% on AndroidControl. These results support using relevant rather than randomly selected history, while leaving open the incremental benefit over a controller that receives no historical prior.
Critical weaknesses
- The incremental contribution of retrieved experience remains insufficiently isolated from the controller's learned token-selection ability. The supervision in Eqs. 7 and 12 is derived from candidate masks containing an annotated target region. This gives the controller a direct localization-related learning signal. It does not imply that the controller ignores history, but it makes a comparison with a controller-only baseline important. Reporting how often each candidate supplies the target mask would also help characterize the supervision; the construction alone does not establish which candidate is typically selected.
Random-History Retrieval provides relevant evidence, but it compares useful history with potentially misleading history. It does not establish the gain over having no historical prior. The page-template states in Appendix A.2 offer a useful descriptive analysis: the controller can run with a zero spatial prior, and Table 5 reports 1,576 such steps at 2,197.2 tokens per decision, compared with 2,550.0 for the no-memory category and 1,913.4 for mask-bearing records. Accuracy on these strata would be informative, but differences in their screen and task distributions prevent this comparison from isolating the effect of the prior.
A stronger test would compare relevant, zero, and shuffled priors on the same held-out states, using the same controller and matched retained-token budgets. This would measure the existing controller's dependence on the prior. A separately trained controller without historical inputs, using the same training data and comparable capacity, would address whether similar accuracy–efficiency trade-offs can be achieved without experience retrieval. Distinguishing these two questions would substantially strengthen the central attribution claim.
- Robustness to stale experience is not established. PAVE uses retrieved masks as inputs to a controller that also observes the current screen; it does not simply apply an old mask unchanged. Nevertheless, a previously useful spatial prior could become misleading after a layout, theme, or application-version change. The current evaluation does not establish how reliably the retrieval gate rejects such records, or how effectively the controller compensates when stale records are retrieved.
AndroidWorld's application selection and execution protocol impose constraints for reproducible evaluation, including local data storage, no required authentication, and fixed application versions (AndroidWorld, Section 3.3 and Appendix C.1). This limits what the benchmark establishes about deployment beyond those conditions. In particular, a fixed-version evaluation does not test whether persistent experience remains useful across application updates. That omission is directly relevant to PAVE's persistence claim. A controlled perturbation study, together with a description of any record invalidation or refresh policy, would address this concern. Describing these experiments as “in-the-wild” would overstate the deployment conditions examined.
Additional concerns, secondary to the two points above:
(a) The threshold sweep would benefit from a decomposition of its effects. Changing the similarity threshold can affect both the fraction of states that receive historical conditioning and the composition of the retrieved records. The reported accuracy ranges are 42.28–43.47% on GUIOdyssey and 66.81–66.91% on AndroidControl, corresponding to ranges of 1.19 and 0.10 percentage points, respectively. The cited token-reduction range of 2.27–17.40% is substantial, but these aggregate results do not identify the contribution of retrieval coverage versus selection quality. Please report, for each threshold, the conditioning rate, the number of mask-bearing records used, reduction on conditioned states, and uncertainty in accuracy. Without these quantities, the sweep supports an operating trade-off more directly than it explains retrieval relevance.
(b) Baseline coverage and operating-point comparisons could be stronger. Appendix B describes adaptations of FastV and VisionZip to GUI-Owl. Such adaptations may be necessary, but their departures from the original methods and any implementation checks should be made clear. Matching baselines to PAVE's per-state token budget is a useful control; additional operating points would show whether the comparison holds across accuracy–retention trade-offs. FocusUI is a relevant comparison for efficient UI grounding, subject to differences in training, backbone, resolution, and token accounting. The paper should also clarify its relationship to intra-trajectory efficiency methods and to cross-episode memory methods such as MementoGUI. These methods need not all be direct experimental baselines if their objectives or interfaces differ substantially, but those differences should be explained.
(c) Performance-preservation claims should reflect the uncertainty and evaluation scope. On AndroidWorld, the paired result is 123/255 successes for PAVE versus 125/255 for the backbone, a difference of approximately −0.8 percentage points, with a reported 95% confidence interval of [−3.8, 2.5] and McNemar p = 0.804. This is compatible with similar performance, but also with a practically relevant loss; failure to detect a difference does not establish equivalence. The fixed-bank result, 33/71 versus 34/71, should be interpreted with the same caution. The abstract would be more precise if it described similar observed task success rather than implying established performance equivalence.
The offline evaluation focuses on click grounding, whereas AndroidWorld supplies end-to-end task evidence. The limitation is therefore the breadth and precision of that evidence, rather than a complete absence of task-level validation. A breakdown by action type and task category would help characterize where performance is preserved. Appendix F's runtime comparison is useful, but a breakdown of total per-decision latency and peak memory, including auxiliary components, would make the practical benefit of token reduction easier to assess.
(d) The scope of transfer and the offline–online gap need clearer characterization. App-scoped retrieval can reuse experience across different tasks within the same app; it does not, by itself, demonstrate cross-app transfer. Please specify whether “across tasks” refers to distinct task instances, task templates, or broader transfer. Similarly, the reduction from 14.93%/17.40% offline to 8.29% online could reflect differences in bank coverage, screen distributions, or other protocol factors. More diverse interfaces are a plausible explanation, but a decomposition by coverage and conditioned-state reduction would provide stronger evidence.
(e) Bank maintenance and storage implications need more detail. Appendix E.2 specifies a fixed bank of 477 records, including 351 page templates and 126 sparse masks. Please also report how bank size and retrieval latency evolve during the online runs, and describe any capacity limit, deduplication, eviction, or refresh policy. Storing embeddings instead of raw screenshots changes the retained information but does not by itself establish privacy protection. A brief account of what is stored, where it is stored, and how it can be deleted would help assess the deployment implications.
Questions for authors
-
Contribution of the historical prior. Can you compare relevant, zero, and shuffled priors on the same states with matched retained-token budgets? Please distinguish this inference-time ablation from a controller trained without historical inputs. Accuracy for the zero-spatial-prior page-template stratum in Table 5 would also be useful, but would not alone isolate the prior's contribution. Which candidate in Eq. 12 most often supplies the supervision target?
-
Retrieval coverage. At each threshold, what fraction of decisions is conditioned, how many mask-bearing records are used, and how much pruning occurs on conditioned decisions? Can you report uncertainty in the accuracy differences and distinguish changes in retrieval coverage from changes in selection quality?
-
Comparable efficiency baselines. Can you include a comparison with FocusUI under a compatible setup, or explain the relevant incompatibilities? FocusUI reports a decrease from 48.3% to 45.1% on ScreenSpot-Pro at its nominal 30% retention setting, together with runtime and memory measurements. These results are not directly comparable to PAVE's current benchmarks. A comparison should align the task, backbone, training assumptions, image resolution, and token-accounting conventions as far as possible, and report several operating points.
-
Stale experience. Can you evaluate a bank built before a controlled interface change? On the same perturbed screens, compare a zero prior, an old prior, and an updated prior, with matched token budgets where feasible. Report the normal retrieval pipeline separately, including its conditioning rate, since gate rejection is itself a potential robustness mechanism. Forced retrieval of old records would be a useful additional stress test, but should be distinguished from normal deployment behavior. The experiment should also measure the current-screen controller's sensitivity to the perturbation itself.
-
Full-pipeline cost. Does the runtime accounting in Appendix F include all auxiliary stages? Please report total per-decision latency, peak memory, and a component breakdown for the visual encoders, OCR where used, retrieval, the controller, and action generation. Throughput or FLOPs would be useful where they match the intended deployment setting, but clear end-to-end latency accounting is the first priority.
Suggestions for future work, rather than requirements for this assessment:
(a) Combine cross-episode token selection with compatible intra-trajectory computation reuse and evaluate the joint accuracy–efficiency trade-off.
(b) Report an annotation-assisted reference within a clearly specified candidate family. This could help contextualize achievable savings under that restricted search, without presenting it as the globally smallest sufficient token subset.
7 comments
Q2: Retrieval coverage. At each threshold, what fraction of decisions is conditioned, how many mask-bearing records are used, and how much pruning occurs on conditioned decisions? Can you report uncertainty in the accuracy differences and distinguish changes in retrieval coverage from changes in selection quality?
We appreciate the suggestion to distinguish retrieval coverage from token-selection quality. The retrieval threshold influences both the proportion of decisions receiving historical conditioning and the composition of retrieved experiences. To examine these effects, we report the controller-conditioning rate, number of mask-bearing retrievals, and token reduction on conditioned decisions at each threshold.
| Dataset | Threshold | Conditioned decisions | Mask-bearing retrievals | Reduction on conditioned decisions |
|---|---|---|---|---|
| GUIOdyssey | 0.75 | 93.53% | 1,705 | 19.20% |
| GUIOdyssey | 0.80 | 84.25% | 1,553 | 19.53% |
| GUIOdyssey | 0.85 | 69.64% | 1,339 | 21.00% |
| GUIOdyssey | 0.90 | 48.50% | 1,002 | 23.96% |
| GUIOdyssey | 0.95 | 24.37% | 545 | 26.14% |
| AndroidControl | 0.75 | 72.49% | 2,180 | 23.86% |
| AndroidControl | 0.80 | 54.28% | 1,844 | 26.26% |
| AndroidControl | 0.85 | 36.22% | 1,314 | 28.47% |
| AndroidControl | 0.90 | 19.60% | 790 | 29.61% |
| AndroidControl | 0.95 | 7.22% | 323 | 30.53% |
As the threshold decreases from 0.95 to 0.75, the conditioning rate increases substantially from 24.37% to 93.53% on GUIOdyssey and from 7.22% to 72.49% on AndroidControl, while token reduction on conditioned decisions remains relatively stable (19.20%–26.14% and 23.86%–30.53%, respectively). These results suggest that the greater overall token savings at lower thresholds are primarily driven by broader retrieval coverage rather than more aggressive token pruning on individual decisions.
To assess the reliability of the accuracy differences, we conduct paired accuracy comparisons using state-level bootstrap resampling. The resulting 95% confidence intervals for accuracy differences relative to the selected operating points include zero across all evaluated thresholds, indicating that no statistically significant accuracy differences are detected.
Q3: Comparable efficiency baselines. Can you include a comparison with FocusUI under a compatible setup, or explain the relevant incompatibilities? FocusUI reports a decrease from 48.3% to 45.1% on ScreenSpot-Pro at its nominal 30% retention setting, together with runtime and memory measurements. These results are not directly comparable to PAVE's current benchmarks. A comparison should align the task, backbone, training assumptions, image resolution, and token-accounting conventions as far as possible, and report several operating points.
We thank the reviewer for suggesting FocusUI as an additional baseline. To provide a more comprehensive comparison, we conduct additional experiments comparing PAVE with FocusUI under two operating conditions: (1) comparable visual-token reduction rates and (2) comparable click accuracies.
At similar visual-token reduction rates (14.93% for PAVE vs. 14.55% for FocusUI), PAVE and FocusUI achieve click accuracies of 43.35% and 33.29%, respectively, representing a 30.2% relative improvement over FocusUI. Furthermore, at comparable click accuracies (32.69% for PAVE vs. 32.87% for FocusUI), PAVE achieves a substantially higher visual-token reduction rate (49.76% vs. 24.07%), corresponding to a 106.7% relative improvement. These results demonstrate that PAVE achieves a more favorable accuracy–efficiency trade-off.
We also thank the reviewer for highlighting relevant work. Intra-trajectory efficiency methods primarily exploit temporal redundancy within individual episodes, whereas PAVE leverages persistent cross-episode experience for token selection. Unlike cross-episode memory methods such as MementoGUI, which focus on improving action prediction, PAVE uses historical experience to reduce visual-token computation without modifying the action model. We will further clarify these distinctions in the revised manuscript.
We thank the reviewer for the constructive feedback and insightful questions. Below, we provide detailed responses to address these concerns, and we hope our additional experimental results and clarifications help resolve these concerns. We will also incorporate the new results and corresponding discussions into the revised manuscript. Thank you very much.
Q1: Contribution of the historical prior. Can you compare relevant, zero, and shuffled priors on the same states with matched retained-token budgets? Please distinguish this inference-time ablation from a controller trained without historical inputs. Accuracy for the zero-spatial-prior page-template stratum in Table 5 would also be useful, but would not alone isolate the prior's contribution. Which candidate in Eq. 12 most often supplies the supervision target?
We appreciate the reviewer's suggestion to distinguish the contribution of historical priors. To isolate the effect of historical priors, we propose an additional controlled ablation using the same held-out states, frozen backbone, controller checkpoint, and per-state visual-token budgets. Specifically, we compare three inference-time conditions: (1) PAVE (relevant); (2) zero; and (3) shuffled. We additionally consider a separately trained no-history controller, which receives only the current visual features, token positions, and task embedding, using the same supervision data and comparable model capacity.
As shown in the results, PAVE consistently achieves the best performance across both benchmarks. Compared with the zero-prior setting, incorporating relevant historical spatial priors improves accuracy by 8.68% on GUIOdyssey and 1.82% on AndroidControl, demonstrating the additional benefit of historical experience beyond the controller's current-screen token-selection capability. In contrast, replacing relevant priors with shuffled priors leads to accuracy drops of 2.34% and 1.82%, respectively, highlighting the importance of retrieving relevant rather than arbitrary historical information. Furthermore, PAVE outperforms the separately trained no-history controller by 4.81% and 2.91%, suggesting that incorporating historical experience provides additional benefits beyond those achievable through controller training alone. Regarding the supervision construction in Eq. 12, the candidate family combines the target region with optional OCR, layout, masks, etc. Among the 4,212 selected verified sparse targets, 18.92% are derived from candidates containing the target region with the top and bottom layout bands (Gt∪Ut), 18.87% from candidates containing the target region alone (Gt), and 25.83% from candidates incorporating projected masks from eligible training records.