- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoFocus, a training-free test-time refinement framework for GUI grounding in high-resolution interfaces. Its central observation is that the perplexity of autoregressively generated coordinate tokens provides a useful proxy for localization uncertainty. AutoFocus first uses visual self-verification to determine whether the initial prediction requires refinement. For selected cases, it samples coordinate hypotheses and maps their horizontal and vertical perplexities into anisotropic Gaussian kernels. The resulting spatial density field guides global and local region proposals, which are refined through Shape-Aware Zooming. Finally, a multi-image visual aggregation step selects the final grounding prediction. Experiments on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G demonstrate consistent overall improvements across general-purpose and GUI-specialized VLMs. On ScreenSpot-Pro, AutoFocus improves Qwen2.5-VL-72B from 47.8% to 65.1% and UI-Venus2, a state-of-the-art GUI agent built on Qwen3.5, from 72.2% to 75.6%, demonstrating its effectiveness even on highly capable contemporary GUI models.
Critical strengths
-
An interesting and useful connection between generative uncertainty and spatial search. The main contribution is the conversion of coordinate-token perplexity into a spatial representation that actively guides visual refinement. Figure 2 provides consistent evidence across four models that incorrect predictions tend to have higher perplexity. While perplexity is an imperfect uncertainty proxy, the paper shows that it can nevertheless support an effective grounding strategy. This is a meaningful contribution to uncertainty-aware visual reasoning.
-
Practical applicability without additional training. AutoFocus requires no additional annotations, reward modeling, or parameter updates. Its improvements across Qwen2.5-VL, UI-TARS, GTA1-Qwen, UI-Venus, and UI-Venus2 demonstrate applicability to models with different grounding capabilities and training histories. The extension to the Qwen3-VL-based UI-Venus and Qwen3.5-based UI-Venus2 also broadens the empirical evidence beyond Qwen2.5-VL-based models. For models exposing coordinate-token likelihoods and supporting the required visual reasoning, the framework offers a practical way to improve an existing inference pipeline.
-
Broad and convincing empirical improvements, including on a strong contemporary GUI agent. The evaluation covers challenging professional interfaces, general mobile and web interfaces, and diverse step-level grounding tasks. The gains are substantial on ScreenSpot-Pro and remain positive on ScreenSpot-V2 even when baseline accuracy is already high.
Critical weaknesses
-
Make the deployment implications of the accuracy–latency trade-off more explicit. The latency analysis is a valuable addition and resolves the initial concern about missing runtime measurements. Under the default setting, however, AutoFocus increases average runtime from 3.26 to 14.91 seconds per example, approximately a 4.6× increase. This is a reasonable trade-off for some accuracy-sensitive applications, but its suitability depends on the interaction setting. The paper would be more useful to practitioners if it highlighted both an accuracy-oriented configuration and a lower-latency configuration, with guidance on when each is appropriate.
-
Further characterize the transferability of the uncertainty signal and refinement decisions. The cross-model evidence, including the new UI-Venus2 results, and the additional analyses described in the author response strengthen the use of perplexity as an uncertainty proxy. A useful next step would be to explain more clearly how its behavior changes across interface domains and failure types. In particular, errors caused by insufficient visual resolution may respond differently to refinement than errors caused by semantic confusion or confidently incorrect predictions.
-
Consolidate implementation and comparison details for reproducibility. The appendix provides useful settings, including model-dependent exploration scales. The final version would benefit from a compact implementation summary specifying verifier and aggregator checkpoints, prompts, coordinate-token extraction, proposal filtering, and the procedure used to select hyperparameters.
Questions for authors
-
Could the final version identify a recommended lower-latency configuration alongside the default setting?
-
Building on the existing uncertainty and verifier analyses, could you summarize which error types benefit most from refinement and which remain difficult to recover from?
-
Could you provide a consolidated description of the verifier and aggregator configurations and the hyperparameter-selection procedure, clarifying whether hyperparameters were selected on a separate validation split and whether reasoning models such as UI-Venus2 required model-specific adjustments?
Reviewers P75E & iWLW] Urgent Reminder: Discussion Period Ends November 18
Dear Reviewers P75E & iWLW,
The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.
Reviewer QBsy already responded and reconsider the rating of this submission.
If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.
Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.
Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.
Thank you for your cooperation.