- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AQuaUI, a training-free method for reducing visual tokens in screenshot-based GUI agents. It partitions each screenshot with an adaptive quadtree, retains one representative token per leaf, and preserves that token’s original spatial position in the model’s positional encodings. For multi-step interactions, a conditional refinement mechanism uses preceding screenshots to retain finer partitions in static or shifted regions. Experiments are performed on multiple datasets and vllms.
Critical strengths
The quadtree method of finding the representative content in the GUI screenshot is interesting.
Critical weaknesses
- The results in Tables 1 and 2 suggest some challenges regarding the practicality of the proposed method. On the one hand, there are cases where the method leads to a substantial drop in accuracy while providing only marginal latency reduction. For example, under the Qwen2-VL-7B-Instruct vLLM setting on the ScreenSpot-V2 dataset, the accuracy drops by 10.46%, whereas the latency is reduced by only 0.02. On the MMB-GUI dataset, the accuracy decreases by 10.57%, but the latency is reduced by only 0.03. On the other hand, for ShowUI-2B under the vLLM setting, +AQuaUI consistently shows significantly higher latency, while still suffering from an accuracy decline. Similar observations can also be found in Table 2. These results raise concerns about the practical effectiveness of the proposed method.
- The proposed method involves many hyperparameters. How were these hyperparameters selected? How sensitive is the model to these choices? Do different models or datasets require separate hyperparameter tuning? For example, this includes the splitting criterion parameter α, the static mode threshold τ_static, the shifted mode threshold τ_shift, γ, etc.
Questions for authors
Please see the weakness section
3 comments
W2. Hyperparameters.
- Alpha. Alpha was chosen for a target reduction of about 30% on Qwen3-VL-8B, and accuracy changes by less than 1 point between 1 and 64 on all three benchmarks (table below). Some models prefer a different value. MAI-UI-8B is one: alpha=1 recovers most of its SS-Pro drop (62.56 vs dense 64.14, against 59.27 at alpha=64). We use one alpha for all models and benchmarks in the paper for a fair comparison; in practice, alpha can be tuned per model, and find the balance point between the accuracy and latency tradeoff.
| alpha | 1 | 16 | 64 (paper) | 256 |
|---|---|---|---|---|
| SS-Pro | 55.66 | 55.79 | 55.03 | 53.76 |
| OSWorld-G | 59.86 | 60.39 | 59.50 | 59.50 |
| MMB-GUI | 83.83 | 83.70 | 83.14 | 83.08 |
- Conditional thresholds (tau_static, tau_shift, gamma). On AndroidControl, changing them from (0.97, 0.94, 0.03) to a loose (0.95, 0.90, 0.01) or strict (0.99, 0.97, 0.05) setting changes accuracy by +0.17 and +0.46, so they do not need tuning per dataset.
- Other quadtree parameters. The block size is fixed by the model's token size (28 px for Qwen2-VL, 32 px for Qwen3-VL), the smallest leaf is one token, and the maximum depth is not binding in practice. The prior's shift search range (4 blocks) was not tuned.
Dear Reviewer izUM,
Thank you for your thoughtful and constructive reviews, and for the time that you put into them. Your comments have improved this work in many places. We have replied to every point. We are glad to answer anything that remains unclear, and if you find our replies satisfactory, we would be grateful if you would consider raising your score.
With appreciation,
The authors
We thank the reviewer for the review, and we are glad the reviewer finds the quadtree approach to locating representative content in GUI screenshots interesting. Below we treat latency and accuracy separately (W1), then the hyperparameters (W2).
W1. Practicality: accuracy drops with little latency gain on Qwen2-VL and ShowUI.
There are two separate issues.
Latency. The small or negative gains came from our initial version quadtree implementation, which was slow Python. We vectorized it and run it in parallel with the HF image processor. Outputs are bit-identical to the old code. The remaining overhead is about 6 ms per 4K image. Re-measured on the same hardware (RTX 6000 Ada, 100 requests x 3 repetitions) gives the following table:
Repetitions vary by about 2 to 5%. KV-cache size is proportional to the LLM's visual tokens and drops by 26 to 31%; for 8B on SS-Pro that is 842 MB to 614 MB per request. With 32 concurrent requests, throughput rises by 48% for 8B and 47% for 2B. Gains are small only for tiny models on low-resolution inputs. For example, ShowUI-2B on SS-V2 has a prefill of 0.17 s, and there AQuaUI is +6% at concurrency 1 but -7% at concurrency 32.
Accuracy on Qwen2-VL. We agree that Qwen2-VL loses much more than Qwen3-VL. We checked that the loss is not an implementation error: with no token dropped, pixels and positions match the dense model exactly, and the dropped blocks are almost blank (about 2% of the image's gradient energy). The loss comes from removing tokens from a model that was not trained with sparse inputs. Two things can improve the performance on Qwen2-VL: 1. applying the reduction after the ViT, which raises SS-V2 from 56.2 to 59.8; and 2. using a smaller alpha. At equal compute, though, the two placements are tied (59.3 vs 59.8), and pre-ViT gives three times the latency reduction. We can tune the alpha specifically to cater to the needs of different latency or performance. Moreover, AQuaUI still gives the optimal compression on Qwen2-VL series. AQuaUI drops tokens before the ViT and ShowUI's implementation only mask inside the LLM. Pruning before the ViT skips the ViT's own compute, which is 29 to 48% of the dense prefill on these benchmarks. AQuaUI still has better or on par performance.