- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to reduce repeated processing of historical screenshots by GUI agents. Before visual encoding, PACT-GUI replaces each historical screenshot with a global thumbnail and a local crop anchored to the executed action, while retaining the current screen at its original resolution. It also uses template-based structured decoding. The method requires no training. Across four benchmarks of offline next-action prediction, mean computation falls to approximately 1/6.62 of the baseline. On AndroidControl-Curated, the full method improves step success from 55.20% to 63.90% and reduces per-instance inference time from 5.540 to 3.092 seconds.
Critical strengths
- Compressing history before visual encoding while retaining global layout and action-local detail is simple and training-free. It directly reduces visual processing that later token or KV compression cannot eliminate, giving the method practical value.
- Table 2 measures History Packets and structured decoding separately and supports their acceleration of different stages. History Packets alone improve AndroidControl step success from 55.20% to 59.80%, and the complete system achieves measured inference speedups. This provides evidence on both computation and measured runtime.
Critical weaknesses
- Action anchoring is the central design choice, but the ablations compare thumbnails, local crops, and their combination without directly comparing other region-selection strategies at the same budget. The results therefore do not establish that action information itself improves on a generic global–local representation, limiting the evidence for the core methodological contribution.
- Main-table quality comparisons combine differences in compression and decoding. Structured decoding alone outperforms the full system on AndroidControl and AITW, while compression baselines have not been compared under a shared decoding protocol. The full system's quality advantage therefore does not directly establish that its history representation is better than existing compression methods.
Questions for authors
- With image sizes, token budgets, and decoding fixed, how does action anchoring compare with center crops or other region-selection strategies?
- Does the relative quality–cost ranking persist when all compression baselines use the same structured decoding?
3 comments
W2Q2: Decoding fairness and comparison under a shared SFD protocol
We thank you for highlighting the importance of fair comparisons in decoding. In the original Table 1 in paper, all baseline methods were evaluated within our unified evaluation framework using the same Qwen3-VL backbone and standard autoregressive decoding, without SFD. This design reflects the fact that PACT-GUI is an end-to-end acceleration framework comprising two complementary components: Action-Conditioned History Packets (HP) and Structured Fast Decoding (SFD). Accordingly, Table 1 is intended to compare PACT-GUI as a complete end-to-end system against the corresponding baseline systems, while Table 2 separately evaluates the contributions of HP, SFD, and their combination. Nevertheless, we acknowledge that the end-to-end comparisons in Table 1 alone do not establish the superiority of HP over alternative compression methods under identical decoding settings. In particular, the fact that SFD-only performs better than HP+SFD on AndroidControl and AITW in Table 2 makes such a controlled comparison necessary.
We therefore add a shared-decoder experiment to explicitly isolate the effect of the decoding protocol. Specifically, we apply the same SFD decoding protocol to all baselines and PACT-GUI while keeping all other evaluation settings unchanged. Given the rebuttal timeline, we conduct this additional evaluation on AndroidControl-Curated High, AITW, and Mind2Web. The results are shown below.
| Method | AndroidControl SR ↑ | AndroidControl TFLOPs ↓ | AITW Avg. Acc. ↑ | AITW TFLOPs ↓ | Mind2Web SR ↑ | Mind2Web TFLOPs ↓ | Avg. Acc. ↑ | Avg. TFLOPs ↓ |
|---|---|---|---|---|---|---|---|---|
| GUI-KV | 59.30 | 184.88 | 48.94 | 37.81 | 47.52 | 52.12 | 51.92 | 91.60 |
| ST-Lite | 59.00 | 184.61 | 48.40 | 37.53 | 46.13 | 51.86 | 51.18 | 91.33 |
| HistPrune | 58.90 | 97.18 | 47.83 | 21.08 | 41.85 | 27.00 | 49.53 | 48.42 |
| GUIPruner | 57.50 | 42.45 | 47.25 | 10.68 | 40.26 | 13.24 | 48.34 | 22.12 |
| FastV | 59.10 | 131.82 | 47.60 | 25.53 | 45.12 | 36.53 | 50.61 | 64.63 |
| SparseVLM | 57.80 | 108.71 | 48.01 | 25.75 | 43.87 | 32.96 | 49.89 | 55.81 |
| LLaVA-PruMerge | 32.50 | 92.29 | 44.46 | 22.16 | 14.29 | 28.09 | 30.42 | 47.51 |
| DivPrune | 43.60 | 91.71 | 44.83 | 22.87 | 16.69 | 28.19 | 35.04 | 47.59 |
| PACT-GUI | 63.90 | 55.72 | 50.95 | 14.94 | 44.44 | 17.23 | 53.10 | 29.30 |
The results show that, under the shared SFD protocol, PACT-GUI still achieves the highest average accuracy (53.10), outperforming the strongest baseline, GUI-KV (51.92), and other history-compression methods, e.g., HistPrune (49.53).
In the revised manuscript, we will explicitly state the decoding protocol used by each baseline in Table 1 and include this shared-SFD comparison in the main paper or appendix. These results demonstrate that PACT-GUI’s gains are not solely attributable to SFD: under the same structured decoder, HP still provides a consistent overall benefit.
Thank you for your detailed responses. After considering your rebuttal, I have decided to maintain my original score.
We sincerely thank you for the constructive comments and valuable suggestions. We address each concern in detail below.
W1Q1: Controlled Comparison of Crop-Selection Strategies
Thank you for highlighting this question. To isolate the effect of region selection, we compared action-anchored crops with center, random, and interface-change-based crops on AndroidControl. The interface-change policy selects a window with high pixel differences between the historical screenshot and its successor. For each historical screenshot, all four strategies use the same crop dimensions and output resolution, and thus the same estimated local visual-token budget; the global thumbnail, model, and decoding settings are also fixed. The strategies differ only in the location of the local crop.
We first used the compact configuration in the paper, which was designed to prioritize acceleration. For example, given a 1080 × 2400 screenshot, a crop based on an action point and not clipped by the image boundary covers approximately 238 × 238 pixels, or 2.19% of the original image area. The crop is then resized to 224 × 224 pixels. Under this setting, action anchoring achieves a success rate of 59.7%, matching random cropping and outperforming center cropping by 0.2 percentage points, but falling 0.2 points below interface-change-based cropping. Thus, the original configuration does not establish that action anchoring outperforms all alternative strategies.
We hypothesized that the small local region retained by the original configuration might limit the contextual information available to each crop and make the benefit of action anchoring harder to observe. To investigate this possibility, we progressively increased the crop coverage using two additional configurations, Large and Huge. For the example screenshot and action-point crop described above, Large expands the region selected from the original image to approximately 432 × 432 pixels, or 7.20% of its area; Huge expands it to approximately 724 × 724 pixels, or 20.22% of its area. The resulting crops are then resized to 448 × 448 and 704 × 704 pixels, respectively. Actual crop coverage varies with the action type, action bounding box, and crop location; these values illustrate the scale of the three configurations. The size configuration is:
The SR results of different crop strategy on AndroidControl are shown below.
With greater crop coverage, action anchoring achieves the highest observed success rate in both the Large and Huge configurations, exceeding the strongest alternative in each configuration by 0.2 and 0.3 percentage points, respectively. This provides direct evidence for the effect of action anchoring: under these two configurations, the choice of local region affects the performance of strategies that otherwise share the same multiscale representation. The margins are small, however, and repeated runs or uncertainty estimates would be needed to assess their stability.
When moving from Large to Huge, the success rates of all four strategies decrease. This indicates that retaining more local content does not necessarily improve performance and is consistent with the possibility that excessive context introduces redundancy. The paper uses the more compact crop configuration to prioritize inference acceleration; the additional experiments illustrate the trade-off among crop selection, local context, and efficiency.