acceptodds
Under review as a conference paper at ICLR 2027

AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

Abstract

Large Multimodal Models (LMMs) have recently emerged as promising backbones for Graphical User Interfaces (GUIs) agent models, where high-resolution GUI screenshots are introduced to the prompts at each iteration step with massive amounts of visual tokens. However, these screenshots exhibit highly non-uniform spatial information density: large regions may carry little information and are visually homogeneous, while text and icons require high visual fidelity. Existing approaches to this problem either require additional training or rely on attention-based token compression, ignoring the structured layout and spatial redundancy of GUI screenshots. To fill the gap, this paper proposes __AQuaUI__, a training-free inference-time token reduction method for GUI agent models that utilizes the non-uniform information density in screenshots. AQuaUI constructs an adaptive quadtree on each image and keeps one representative merged token per leaf of the quadtree. It preserves the spatial positions of retained tokens throughout the pipeline to ensure that all position-encoding stages remain consistent. We further propose a conditional quadtree algorithm that utilizes temporal continuity between consecutive screenshots within a single request to refine the current quadtree with previous ones as references to preserve fine-grained regions across static or mildly shifted GUI states. We implement AQuaUI for state-of-the-art GUI agent models on vLLM and conduct experiments on standard grounding and navigational benchmarks. AQuaUI consistently shows improved accuracy-efficiency trade-offs over prior baselines. Notably, on GUI-Owl-1.5-32B-Instruct, AQuaUI reduces average latency by __13.10%__ with __29.52%__ fewer visual tokens while retaining __99.06%__ of full-token performance, suggesting that the spatial redundancy of GUI screenshots can be exploited at inference time without retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agents, Efficient Inference, Quadtree

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_izUMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AQuaUI, a training-free method for reducing visual tokens in screenshot-based GUI agents. It partitions each screenshot with an adaptive quadtree, retains one representative token per leaf, and preserves that token’s original spatial position in the model’s positional encodings. For multi-step interactions, a conditional refinement mechanism uses preceding screenshots to retain finer partitions in static or shifted regions. Experiments are performed on multiple datasets and vllms.

Critical strengths

The quadtree method of finding the representative content in the GUI screenshot is interesting.

Critical weaknesses
  1. The results in Tables 1 and 2 suggest some challenges regarding the practicality of the proposed method. On the one hand, there are cases where the method leads to a substantial drop in accuracy while providing only marginal latency reduction. For example, under the Qwen2-VL-7B-Instruct vLLM setting on the ScreenSpot-V2 dataset, the accuracy drops by 10.46%, whereas the latency is reduced by only 0.02. On the MMB-GUI dataset, the accuracy decreases by 10.57%, but the latency is reduced by only 0.03. On the other hand, for ShowUI-2B under the vLLM setting, +AQuaUI consistently shows significantly higher latency, while still suffering from an accuracy decline. Similar observations can also be found in Table 2. These results raise concerns about the practical effectiveness of the proposed method.
  2. The proposed method involves many hyperparameters. How were these hyperparameters selected? How sensitive is the model to these choices? Do different models or datasets require separate hyperparameter tuning? For example, this includes the splitting criterion parameter α, the static mode threshold τ_static, the shifted mode threshold τ_shift, γ, etc.
Questions for authors

Please see the weakness section

3 comments

We thank the reviewer for the review, and we are glad the reviewer finds the quadtree approach to locating representative content in GUI screenshots interesting. Below we treat latency and accuracy separately (W1), then the hyperparameters (W2).

W1. Practicality: accuracy drops with little latency gain on Qwen2-VL and ShowUI.

There are two separate issues.

Latency. The small or negative gains came from our initial version quadtree implementation, which was slow Python. We vectorized it and run it in parallel with the HF image processor. Outputs are bit-identical to the old code. The remaining overhead is about 6 ms per 4K image. Re-measured on the same hardware (RTX 6000 Ada, 100 requests x 3 repetitions) gives the following table:

Change vs dense (latency, SS-Pro)old codenew code
Qwen3-VL-8B, 1 / 3 / 5 / 10 screenshots-1% / +11% / +20% / +20%-20% / -24% / -23% / -24%
Qwen3-VL-2B, 1 / 5 screenshots+13% / +43%-18% / -21%
Qwen3-VL-32B, 1 screenshot-6%-20%
Qwen2-VL-7B, SS-V2, 1 screenshot+2%-8%
ShowUI-2B, SS-V2, concurrency 1 / 32+30% / +11%+6% / -7%

Repetitions vary by about 2 to 5%. KV-cache size is proportional to the LLM's visual tokens and drops by 26 to 31%; for 8B on SS-Pro that is 842 MB to 614 MB per request. With 32 concurrent requests, throughput rises by 48% for 8B and 47% for 2B. Gains are small only for tiny models on low-resolution inputs. For example, ShowUI-2B on SS-V2 has a prefill of 0.17 s, and there AQuaUI is +6% at concurrency 1 but -7% at concurrency 32.

Accuracy on Qwen2-VL. We agree that Qwen2-VL loses much more than Qwen3-VL. We checked that the loss is not an implementation error: with no token dropped, pixels and positions match the dense model exactly, and the dropped blocks are almost blank (about 2% of the image's gradient energy). The loss comes from removing tokens from a model that was not trained with sparse inputs. Two things can improve the performance on Qwen2-VL: 1. applying the reduction after the ViT, which raises SS-V2 from 56.2 to 59.8; and 2. using a smaller alpha. At equal compute, though, the two placements are tied (59.3 vs 59.8), and pre-ViT gives three times the latency reduction. We can tune the alpha specifically to cater to the needs of different latency or performance. Moreover, AQuaUI still gives the optimal compression on Qwen2-VL series. AQuaUI drops tokens before the ViT and ShowUI's implementation only mask inside the LLM. Pruning before the ViT skips the ViT's own compute, which is 29 to 48% of the dense prefill on these benchmarks. AQuaUI still has better or on par performance.

W2. Hyperparameters.

  • Alpha. Alpha was chosen for a target reduction of about 30% on Qwen3-VL-8B, and accuracy changes by less than 1 point between 1 and 64 on all three benchmarks (table below). Some models prefer a different value. MAI-UI-8B is one: alpha=1 recovers most of its SS-Pro drop (62.56 vs dense 64.14, against 59.27 at alpha=64). We use one alpha for all models and benchmarks in the paper for a fair comparison; in practice, alpha can be tuned per model, and find the balance point between the accuracy and latency tradeoff.
alpha11664 (paper)256
SS-Pro55.6655.7955.0353.76
OSWorld-G59.8660.3959.5059.50
MMB-GUI83.8383.7083.1483.08
  • Conditional thresholds (tau_static, tau_shift, gamma). On AndroidControl, changing them from (0.97, 0.94, 0.03) to a loose (0.95, 0.90, 0.01) or strict (0.99, 0.97, 0.05) setting changes accuracy by +0.17 and +0.46, so they do not need tuning per dataset.
  • Other quadtree parameters. The block size is fixed by the model's token size (28 px for Qwen2-VL, 32 px for Qwen3-VL), the smallest leaf is one token, and the maximum depth is not binding in practice. The prior's shift search range (4 blocks) was not tuned.

Dear Reviewer izUM,

Thank you for your thoughtful and constructive reviews, and for the time that you put into them. Your comments have improved this work in many places. We have replied to every point. We are glad to answer anything that remains unclear, and if you find our replies satisfactory, we would be grateful if you would consider raising your score.

With appreciation,

The authors

Reviewer_TfFxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AQuaUI reduces visual tokens for GUI agents without training additional parameters. It builds adaptive quadtrees on the native merged-token grid, retains a central block per leaf, and preserves original coordinates through vision- and language-side positional encodings. The default implementation prunes before the vision encoder; a conditional variant refines partitions using preceding screenshots. Result show consistency such as GUI-Owl-1.5-32B reported 29.52% token reduction, 13.10% average latency reduction, and 99.06% retention of macro-average accuracy.

Critical strengths
  1. The method addresses a practical bottleneck with a transparent image prior and no additional training.
  2. Results show Several larger Qwen3-based checkpoints preserve most grounding accuracy while reducing latency.
  3. The paper discloses important limitations: overhead on smaller models, weaker Qwen2 results, pre-/post-ViT differences, threshold sensitivity, and context-loss failures.
Critical weaknesses
  1. The hardware, precision, resolution, decoding, batching, and cache settings underlying the main tables are not fully specified.
  2. The conditional-history configuration is unspecified and paired outcomes from existing trajectories are not reported.
Questions for authors
  1. What exact hardware, precision, resolution, decoding, batching, and cache settings underlie the main tables? Were they identical within each dense/AQuaUI pair, and how are padding tokens counted?
  2. Can the authors specify the conditional-history configuration and report paired outcomes from existing trajectories, while narrowing robustness claims to the single-seed and input-sensitivity evidence currently available?
  3. How was alpha=64 selected, how are DeepStack and positional indices aligned after pruning, and which model/interface regimes do the authors consider supported given the disclosed failures?
4 comments

We thank the reviewer for the careful reading, including the limitations we disclosed in the appendix. We are glad the reviewer values the training-free design with a transparent image prior, and that the larger Qwen3-VL checkpoints keep most of their accuracy while running faster. Below we give the settings that were missing, report new paired results for the conditional prior, and narrow the claims where the evidence is thin, as suggested.

W1&Q1. Settings.

  • Hardware and precision. 4x RTX 6000 Ada (48 GB, PCIe), vLLM 0.12.0, bf16. Tensor parallel size is 2 for 2B and 8B models and 4 for 32B models.
  • Resolution. The image processor's default range (Qwen3-VL: up to 16.7 MP), so screenshots are processed at native resolution.
  • Decoding. Temperature 0.1, top-p 1.0, max 512 new tokens for Qwen2-VL, Qwen3-VL and MAI-UI; temperature 0.01, top-k 1 for GUI-Owl.
  • Batching. Two client threads for 2B and 8B models, so at most two requests are in flight, with vLLM's default continuous batching.
  • Cache. vLLM defaults (prefix caching and multimodal processor cache on).
  • Pairing. All of these were identical within each dense/AQuaUI pair.
  • Padding. Padding tokens are counted in the token and compression numbers. 44 to 49% of images need padding, 2.2 tokens on average and at most 9.

For the Qwen3-VL and MAI-UI tables, the temperature is set to 0.1. To show the sampling does not affect conclusions, we repeated the SS-Pro runs:

DenseAQuaUI
3 runs (T=0.1)56.10 / 55.91 / 56.6155.03 / 55.15 / 55.15
Greedy56.0455.03

The gap is stable at about 1 point. On OSWorld-G the two are within 0.5 points in every run.

W2&Q2. Conditional-history configuration and paired outcomes.

  • Configuration. The prior links each screenshot to the one right before it in the same request. The first screenshot of a request is decomposed on its own, nothing is carried across requests, and the prior is skipped when the two screenshots differ in resolution. Both frames are reduced to one grayscale value per token-sized block, and similarity is one minus the mean absolute difference of these values (scaled to 0 to 1). A region of the current frame reuses the previous partition when its similarity is at least tau_static=0.97; otherwise, if a shift of up to 4 blocks reaches tau_shift=0.94 and beats no shift by gamma=0.03, it reuses the shifted partition; otherwise it is decomposed from scratch. The prior only splits leaves and never merges them, so it can only add tokens. Alpha, block size and position handling are the same as for single images. vLLM's multimodal processor cache must be off for the prior to run: with it on, the earlier frame is often served from the cache and the processor sees only one image. All conditional runs below use --mm-processor-cache-gb 0.
  • Paired outcomes. For AndroidWorld we kept only per-arm success rates, so paired outcomes are not available there. Instead, we replayed the recorded AndroidControl-High trajectories, so both arms see the same screens at every step, and compared the arms step by step (Qwen3-VL-8B):
AndroidControl-Highstepsno priorwith priorsteps only one arm gets right (no prior / prior)
2 screenshots6,29660.6760.59127 / 122
5 screenshots2,78555.4055.0169 / 58
2 screenshots, alpha=1000 (about 60% of tokens)6,29660.3260.51110 / 122
2 screenshots, 1.4 MP cap + alpha=1000 (about 32% of tokens)6,29658.9959.01146 / 147
2 screenshots, random extra tokens in place of the prior6,29660.6760.85121 / 132
5 screenshots, random extra tokens in place of the prior2,78555.4054.8367 / 51

The prior changes the answer on about 4% of steps, about equally often in each direction, and spending the same number of extra tokens at random positions does the same. As you suggest, we narrow the claims. Table 4 will be described as a single run (seed 30), and we drop the statement that the prior improves success. Contribution 2 will center on AQuaUI for multi-screenshot inputs, where accuracy is unchanged and per-step latency drops by 23%; the prior becomes a mechanism that keeps token layouts consistent across frames, with no measurable effect on accuracy. Robustness claims for the prior will rest only on the threshold sensitivity: loose (0.95, 0.90, 0.01) and strict (0.99, 0.97, 0.05) settings change accuracy by +0.17 and +0.46.

Q3. Alpha, DeepStack and position alignment, supported regimes.

Alpha. Alpha was set to reach about 30% reduction on screenshots on Qwen3-VL-8B, before looking at accuracy. Between alpha=1 and 64, accuracy changes by less than 1 point on all three benchmarks. We use one alpha for all models and benchmarks in the paper for a fair comparison; in practice, alpha can be tuned per model.

alpha11664 (paper)256
SS-Pro55.6655.7955.0353.76
OSWorld-G59.8660.3959.5059.50
MMB-GUI83.8383.7083.1483.08

DeepStack and positions. DeepStack features are taken from intermediate ViT layers on the same token sequence and concatenated along the feature dimension, so they are pruned together with the final tokens. Each kept token is the center block of its leaf and keeps that block's (row, column) in the dense grid. The ViT's 2D rotary embedding, Qwen3-VL's interpolated absolute position embedding and the LLM's M-RoPE all use these coordinates, so a kept token gets exactly the position it has in the dense sequence. To test the whole path end to end, we kept every token but routed them through AQuaUI's selection and position-injection path. With greedy decoding this matches dense (SS-Pro 55.98 vs 56.04, OSWorld-G 59.15 vs 59.33). So the pipeline itself is lossless, and any accuracy change comes from the dropped tokens.

Supported regimes. Qwen2-VL and Qwen3-VL families on desktop and web screenshots where inputs are large can have the best performance. On mobile settings, AQuaUI would still work but on par with other compression methods. Qwen2.5-VL (windowed ViT attention) may struggle due to its special ViT attention design. We will state this in the limitations.

Dear Reviewer TfFx,

Thank you for your thoughtful and constructive reviews, and for the time that you put into them. Your comments have improved this work in many places. We have replied to every point. We are glad to answer anything that remains unclear.

With appreciation,

The authors

Reviewer_ZvD6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents aquaui, which reduces visual tokens in gui agents without extra training, uses quadtrees to select tokens while keeping their original positions, and uses earlier screenshots to refine selection, with experiments on grounding and navigation.

Critical strengths

The method addresses a practical problem, reducing visual tokens by about 30% and average latency by about 13% on gui owl 32b, with a small average accuracy drop.

Critical weaknesses
  1. The contribution seems incremental, and the experiments do not clearly show what the quadtree adds over simpler structured token selection with the same positional handling
  2. The online navigation results show fewer tokens but lower success rates than the full-token baseline, and total task time or compute cost is not reported, so the practical benefit for agents remains unclear
Questions for authors

Could you compare the quadtree with simpler structured token selection under the same setup to clarify the value of this design? And on the navigation tasks, how does the total runtime or compute cost change alongside task success?

3 comments

We thank the reviewer for the very detailed review. We are glad the reviewer sees the practical value of cutting about 30% of the visual tokens with a small accuracy drop. We appreciate the constructive feedback and address each concern below.

W1&Q1. What does the quadtree add over simpler structured selection with the same position handling?

We added stride sampling, uniform 2x2 merging and ShowUI-style UI-graph selection. All use the same position path as AQuaUI and the same per-image token budget. Like AQuaUI, all of them select tokens before the ViT.

Qwen3-VL-8BSS-ProOSWorld-GMMB-GUIUI-Vision
Dense56.1059.8684.2027.82
AQuaUI55.0359.5083.1427.78
Downsampling (lower max_pixels)55.6657.5583.5026.61
UI-graph (ShowUI-style, pre-ViT)54.6557.5583.0027.21
Stride sampling42.3852.0472.4519.86
Uniform 2x2 merge42.6951.5173.8519.88
Random dropping39.4753.8272.6819.93

Content-agnostic selection (stride, uniform merge, and random) loses 6 to 16 points on every benchmark. Among the content-aware methods, AQuaUI is ahead of UI-graph on all four benchmarks at this budget.

The comparison with simple downsampling is interesting. In our experiments, we found that on the 81 SS-Pro screenshots above 9 MP, downsampling does better than both AQuaUI and dense (48.2 vs 25.9 and 33.3), probably because Qwen3-VL natively handles such large inputs poorly. On desktop screenshots up to 1440p, AQuaUI is ahead of downsampling: +1.1 points on the 6,174 screenshots up to 1080p and +0.8 at 1440p, while at 4K the two are close (-1.2). On smaller phone screenshots, downsampling is as good as AQuaUI. On AndroidControl at the same budget, downsampling scores 56.63 and AQuaUI 56.55 (dense 57.12). Mobile UI elements are large, so they survive downsampling.

In fact, downsampling and AQuaUI remove different redundancy: downsampling removes resolution the model does not need, and AQuaUI removes flat regions at any resolution. They can be combined, and this can be especially useful for large screenshots under tight budget. The following table shows the potential of combining these 2 methods. Split by image size at the same 48.5% budget (4.19 MP cap + AQuaUI vs a 2.91 MP cap):

SS-Pro, 48.5% of dense tokens, max_pixels + AQuaUI vs max_pixels aloneimages <= 4.2 MP (n=820)images > 4.2 MP (n=761)
Qwen3-VL-8B+0.7+4.3
GUI-Owl-1.5-8B+2.0+5.7

We will add the full accuracy-token curves to the paper.

On whether the contribution is incremental: the split rule is deliberately simple. What we see as new is the path around it. Tokens are dropped before the ViT, so the ViT's own compute shrinks too (29 to 48% of the dense prefill on these benchmarks), and every kept token keeps its original 2D position in the ViT's rotary embedding, in Qwen3-VL's interpolated absolute embedding and in the LLM's M-RoPE. With every token kept, this path matches dense accuracy (SS-Pro 55.98 vs 56.04, greedy). The table above shows that once this path is fixed, which tokens are kept still matters: content-agnostic rules lose 6 to 16 points at the same budget, and the quadtree is also ahead of ShowUI's UI-graph.

W2&Q2. Runtime and compute on navigation tasks.

AQuaUI's previous implementation was single-threaded Python and cost up to about 0.2 s per 4K image. We vectorized it and run it in parallel with the HF image processor. Outputs are bit-identical to the old code. The remaining overhead is about 6 ms per 4K image. Re-measured on the same hardware (RTX 6000 Ada, 100 requests x 3 repetitions) gives the following table:

Change vs dense (latency, SS-Pro)old codenew code
Qwen3-VL-8B, 1/ 3 / 5 / 10 screenshots-1% / +11% / +20% / +20%-20% / -24% / -23% / -24%
Qwen3-VL-2B, 1 / 5 screenshots+13% / +43%-18% / -21%
Qwen3-VL-32B, 1 screenshot-6%-20%
Qwen2-VL-7B, SS-V2, 1 screenshot+2%-8%
ShowUI-2B, SS-V2, concurrency 1 / 32+30% / +11%+6% / -7%

Unfortunately, we could not rerun AndroidWorld with timing within the rebuttal period. As a step-level proxy, we evaluated AndroidControl with five screenshots per request, which is the regime an agent with history is in:

AndroidControl-High, 5 screenshotsaccuracyprompt tokenslatency per step
Dense55.2212,7763.78 s
AQuaUI55.409,2612.92 s (-23%)

Accuracy is unchanged while latency and KV cache drop by about a quarter at every step, so the saving accumulates over an episode. Since AQuaUI drops tokens before the ViT, the ViT's compute falls by the same share as the visual tokens. On success rate: on AndroidControl, AQuaUI is 0.2 points above dense with five screenshots (2,785 steps) and 0.5 points below with two (6,296 steps).

Thank you for the thoughtful response and additional experiments. The controlled comparisons, results combining AQuaUI with downsampling, and updated latency measurements strengthen the evidence and address several of my concerns. My remaining reservations concern the modest methodological advance over existing token selection approaches and the lack of full task timing results alongside online navigation success. On balance, I maintain my rating of 2, while appreciating the authors’ efforts to improve the paper.

Reviewer_McPtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AQuaUI, a training-free, inference-time visual token reduction method that adaptively partitions GUI screenshots using a quadtree. It retains only the center token of each leaf node and incorporates the corresponding spatial coordinates into both the ViT and M-RoPE. A conditional quadtree mechanism is further introduced to leverage adjacent screenshots. The authors provide evaluation on Qwen3-VL to demonstrate approximately a 30% reduction in visual tokens, with an average accuracy drop of less than one percentage point. For larger models, the method reduces inference latency by 0.09–0.23 seconds.

Critical strengths

1. Practical, training-free implementation with valuable engineering insights. AQuaUI implements pixel-only, rule-based visual token reduction for VLLMs, circumventing several limitations of existing approaches: GUI-specific token reduction methods that require dedicated training, generic pruning methods that rely on attention maps, and compatibility issues with FlashAttention. Notably, it adapts the ViT-side RoPE and absolute positional interpolation, as well as the LLM-side M-RoPE, to operate on actual spatial coordinates. These implementation details provide practical insights and a valuable reference for the research community.

2. Comprehensive experimental validation on the Qwen3-VL family. The main results are well-supported, covering seven model variants across five grounding benchmarks, providing a reasonably broad evaluation of the proposed method.

3. Transparent analysis of limitations and failure cases. The authors provide an honest assessment of the method's limitations in the appendix, including failure cases involving the loss of local visual details and contextual information. They further report that, among the 343 samples where the dense model succeeds but AQuaUI fails, only eight have target centers falling within merged leaves. This analysis offers useful evidence for understanding the actual sources of performance degradation.

Critical weaknesses
  1. Missing the most direct inference-time baselines; structural effectiveness is weakly supported. The paper does not compare against lowering max_pixels (uniform downsampling) at an equal token budget, which is the simplest way to cut ~30% of tokens on a 2400×1080 phone screenshot and is natively supported by Qwen-VL. Regular grid pooling and stride sampling baselines are also absent. On Qwen3-VL, the only evidence for the benefit of the quadtree structure is random dropping (Table 5), which is not layout-aware, and no GUI-specific baseline (e.g., ShowUI-style selection) is evaluated. ShowUI appears only on Qwen2-VL-7B and ShowUI-2B (Table 1).

  2. The Table 5 baselines are of questionable reliability, and the attribution is confounded. VTW collapses on Qwen3-VL-8B (SS-V2 5.11 vs. 94.26 dense; SS-Pro 0.13 vs. 56.23) and also on Qwen2-VL-7B (e.g., UI-Vision 0.09, MMB-GUI 2.28). This suggests a possible incompatibility between the PACT implementation and these architectures (e.g., DeepStack, M-RoPE) rather than an inherent weakness of the method. The paper does not state whether random dropping preserves the original position coordinates, so the gap to AQuaUI may conflate position preservation with quadtree selection. On Qwen2-VL-7B, random dropping slightly outperforms AQuaUI on SS-V2 (57.47 vs. 56.68) and MMB-GUI (29.60 vs. 29.33), and FastV leads by 0.82–1.57 points on three benchmarks. The text acknowledges the FastV gap but not the random-dropping result. Moreover, random dropping is applied after the vision encoder, while AQuaUI uses its pre-ViT default, so the comparison is confounded, and post-ViT AQuaUI is not compared against the baselines.

  3. The evidence for the conditional quadtree (contribution 2) is thin. The only support is Table 4: AndroidWorld (116 tasks), a single run with seed 30, two models, each showing a gain of 3 tasks (2.6 points). The binomial standard error at this success rate is about 4–5 points, so the difference is indistinguishable from noise. Furthermore, Appendix A.4.2 reports that the prior adds only 16.9 tokens on average (0.95%, measured on 621 AndroidControl screenshot pairs), and the compression gap in Table 4 is only 0.16–0.17 percentage points. A 2.6-point gain from <1% additional tokens requires much stronger evidence than a single run (e.g., multiple seeds with variance, and a control that retains the same number of extra tokens without the prior).

  4. The efficiency and generalization claims outpace the evidence. Latency gains are concentrated on high-resolution inputs such as SS-Pro (Qwen3-VL-32B: 3.24→2.44s); on the other benchmarks, they are mostly in the 0–0.2s range. There is no gain for 2B models, latency increases for Qwen3-VL-2B on AndroidControl (+0.09s), and the 32B model becomes slower on OSW-G (0.97→1.03s). No batch size, number of repetitions, or variance is reported, and there are no measurements of KV-cache memory, throughput, or long-history settings, which are central to the stated motivation. The introduction claims applicability to "any downstream LMM," yet experiments cover only two architecture families (Qwen2-VL and Qwen3-VL; GUI-Owl, MAI-UI, and Qwen3-VL all share the latter), with no Qwen2.5-VL, UI-TARS, or InternVL. Finally, the "<1 point average drop" holds only for the three Qwen3-VL models (MAI-UI-8B: −1.66; GUI-Owl-1.5-8B: −1.01) and masks degradation in high-resolution desktop scenarios, where compression is arguably most needed (SS-Pro: −5.38 for MAI-UI-8B, −2.40 for Qwen3-VL-32B).

Questions for authors

See critical weaknesses

5 comments

We thank the reviewer for the very detailed review. We are glad the reviewer found the position handling (ViT RoPE, absolute position interpolation and M-RoPE on true coordinates) a useful reference, and valued the breadth of the Qwen3-VL experiments and the failure analysis. The suggested baselines and the control for the conditional prior were concrete enough to run directly, and thus we ran them. We address each concern below.

W1. Missing simple baselines at equal token budget.

These baselines are indeed important to include in the paper. We added them, all at AQuaUI's per-image token budget (about 72%) and with the same position-preserving path:

Qwen3-VL-8BSS-ProOSWorld-GMMB-GUIUI-Vision
Dense56.1059.8684.2027.82
AQuaUI55.0359.5083.1427.78
Downsampling (lower max_pixels)55.6657.5583.5026.61
UI-graph (ShowUI-style, pre-ViT)54.6557.5583.0027.21
Stride sampling42.3852.0472.4519.86
Uniform 2x2 merge42.6951.5173.8519.88
Random dropping39.4753.8272.6819.93

Content-agnostic selection (stride, uniform merge, and random) loses 6 to 16 points on every benchmark. Among the content-aware methods, AQuaUI is ahead of UI-graph on all four benchmarks at this budget.

The comparison with simple downsampling is interesting. In our experiments, we found that on the 81 SS-Pro screenshots above 9 MP, downsampling does better than both AQuaUI and dense (48.2 vs 25.9 and 33.3), probably because Qwen3-VL natively handles such large inputs poorly. On desktop screenshots up to 1440p, AQuaUI is ahead of downsampling: +1.1 points on the 6,174 screenshots up to 1080p and +0.8 at 1440p, while at 4K the two are close (-1.2). On smaller phone screenshots, downsampling is as good as AQuaUI. On AndroidControl at the same budget, downsampling scores 56.63 and AQuaUI 56.55 (dense 57.12). Mobile UI elements are large, so they survive downsampling.

In fact, downsampling and AQuaUI remove different redundancy: downsampling removes resolution the model does not need, and AQuaUI removes flat regions at any resolution. They can be combined, and this can be especially useful for large screenshots under tight budget. The following table shows the potential of combining these 2 methods. Split by image size at the same 48.5% budget (4.19 MP cap + AQuaUI vs a 2.91 MP cap):

SS-Pro, 48.5% of dense tokens, max_pixels + AQuaUI vs max_pixels aloneimages <= 4.2 MP (n=820)images > 4.2 MP (n=761)
Qwen3-VL-8B+0.7+4.3
GUI-Owl-1.5-8B+2.0+5.7

We will add the full accuracy-token curves to the paper.

We didn’t compare with ShowUI’s layout-aware method in Table 5 since they have only provided an official implementation on Qwen2-VL series. For this rebuttal, we implemented ShowUI's UI-graph method in pre-ViT token selection for a fair comparison. ShowUI applies its selection to the visual tokens inside the language model, after a full ViT pass, which would give it more compute than AQuaUI at the same token count. Our implemented port runs before the ViT, keeps the same number of tokens per image and uses AQuaUI's position path, so the two differ only in which tokens are kept. AQuaUI still outperforms UI-graph.

W2. Reliability of Table 5.

VTW collapse. We checked whether the collapse is a porting problem by sweeping the withdrawal layer on Qwen3-VL-8B (SS-V2):

Withdrawal layer16 (matched FLOPs)17242832none (dense)
Accuracy4.65.147.194.194.094.1
Prefill FLOPs vs dense68%69%81%87%94%100%

With late withdrawal VTW matches dense exactly. Without withdrawal its logits are identical to the original model. So the implementation is correct. The problem is that matching AQuaUI's compute forces withdrawal at layer 16, below a sharp cliff. On OSWorld-G the matched layer is 25 to 26 because images are small, and there VTW reaches 52 to 58. We will add this sweep and explain it in the text.

Random dropping and position preservation. Random dropping keeps the original coordinates and uses exactly the same position path as AQuaUI; only the choice of tokens differs. We will state this explicitly. On Qwen3-VL-8B random is 15.6 points below AQuaUI on SS-Pro.

The Qwen2 confound. You are right that random was post-ViT and AQuaUI pre-ViT in Table 5. This was an oversight in our experiments. We reran the comparison with both methods in the same placement:

Qwen2-VL-7BSS-V2MMB-GUI
pre-ViT: AQuaUI / random56.21 / 50.8629.30 / 25.82
post-ViT: AQuaUI / random59.75 / 57.1532.47 / 31.44

The 57.47 for random in Table 5 corresponds to the post-ViT row here. AQuaUI beats random in both placements, clearly in pre-ViT and by a smaller margin in post-ViT.

Choice between post- and pre-ViT. Table 5 compared methods at the same number of kept tokens. That is not the same compute, because AQuaUI drops tokens before the ViT and FastV only inside the LLM. Pruning before the ViT also skips the ViT's own compute, which is 29 to 48% of the dense prefill on these benchmarks. Prefill FLOPs (ViT + LLM) relative to dense, on Qwen3-VL-8B:

Qwen3-VL-8BSS-V2SS-ProOSWorld-GUI-VisionMMB-GUI
Visual tokens kept by AQuaUI71%72%82%70%70%
FLOPs, AQuaUI (pre-ViT)67%63%81%68%66%
FLOPs, AQuaUI (post-ViT)82%84%88%82%81%
FLOPs, FastV (K=4)84%86%90%84%83%
Accuracy at equal FLOPs, AQuaUI (pre-ViT) / FastV94.26 / 90.2555.28 / 28.9159.33 / 56.6627.72 / 22.6183.28 / 73.01

To match AQuaUI's compute, FastV can keep only 22 to 67% of the visual tokens, and at that budget AQuaUI is ahead on all five benchmarks.

W3. Evidence for the conditional quadtree.

We agree that a single AndroidWorld run cannot support this claim. We re-tested the prior on AndroidControl, where each request includes the previous screenshot or screenshots. Following your suggestion, we added a control that spends the same extra tokens at random positions. The results are shown below:

AndroidControl-Highno priorconditional priorrandom extra tokensdense
2 screenshots, 6,296 steps60.6760.5960.8561.21
5 screenshots, 2,785 steps55.4055.0154.8355.22

The prior only refines regions that are unchanged from the previous frame, and for those regions the quadtree already gives almost the same partition. In practice it changes the layout in about a quarter of requests and adds 0.4% tokens on average. So the prior keeps token layouts consistent across frames at almost no cost, while accuracy stays the same.

What these experiments do show is that AQuaUI handles the multi-screenshot inputs of agents well:

AndroidControl-High, 5 screenshots per requestdenseAQuaUI
step accuracy55.2255.40
prompt tokens per step12,7769,261 (-28%)
latency per step3.78 s2.92 s (-23%)

History helps on scroll and type steps (+11 and +8 points over a single screenshot), and AQuaUI keeps these gains.

We will revise contribution 2 accordingly. It will describe AQuaUI on multi-screenshot agent inputs, applied per screenshot: it keeps accuracy with history while reducing per-step latency and KV cache by about a quarter.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.