acceptodds
Under review as a conference paper at ICLR 2027

AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding

Abstract

Vision-Language Models (VLMs) have enabled autonomous GUI agents that translate natural language instructions into executable screen coordinates. However, grounding performance degrades in high-resolution interfaces, where dense layouts and small interactive elements expose a resolution gap between modern displays and model input constraints. Recent methods have explored zoom-in refinement, confidence-based selection, and reward-driven optimization. But most of them either rely on manually designed crop policies, consensus over sampled predictions, or additional reinforcement learning objectives. In contrast, we study whether the model’s own autoregressive coordinate distribution can serve as an intrinsic signal for active visual search at test time. We propose AutoFocus, a training-free uncertainty-aware refinement framework that converts token-level coordinate perplexity into an anisotropic spatial probability field. This field guides global-local region proposal generation, while an error-triggered verifier selectively activates refinement only when the initial prediction is uncertain. AutoFocus therefore improves high-resolution GUI grounding without requiring additional annotations, reward modeling, or parameter updates. Extensive experiments on ScreenSpot-Pro, ScreenSpot-V2 and OSWorld-G demonstrate consistent improvements across both general-purpose and GUI-specialized VLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Grounding, Test-time Scaling, Vision and Language

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QBsyOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AutoFocus, a training-free test-time refinement framework for GUI grounding in high-resolution interfaces. Its central observation is that the perplexity of autoregressively generated coordinate tokens provides a useful proxy for localization uncertainty. AutoFocus first uses visual self-verification to determine whether the initial prediction requires refinement. For selected cases, it samples coordinate hypotheses and maps their horizontal and vertical perplexities into anisotropic Gaussian kernels. The resulting spatial density field guides global and local region proposals, which are refined through Shape-Aware Zooming. Finally, a multi-image visual aggregation step selects the final grounding prediction. Experiments on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G demonstrate consistent overall improvements across general-purpose and GUI-specialized VLMs. On ScreenSpot-Pro, AutoFocus improves Qwen2.5-VL-72B from 47.8% to 65.1% and UI-Venus2, a state-of-the-art GUI agent built on Qwen3.5, from 72.2% to 75.6%, demonstrating its effectiveness even on highly capable contemporary GUI models.

Critical strengths
  1. An interesting and useful connection between generative uncertainty and spatial search. The main contribution is the conversion of coordinate-token perplexity into a spatial representation that actively guides visual refinement. Figure 2 provides consistent evidence across four models that incorrect predictions tend to have higher perplexity. While perplexity is an imperfect uncertainty proxy, the paper shows that it can nevertheless support an effective grounding strategy. This is a meaningful contribution to uncertainty-aware visual reasoning.

  2. Practical applicability without additional training. AutoFocus requires no additional annotations, reward modeling, or parameter updates. Its improvements across Qwen2.5-VL, UI-TARS, GTA1-Qwen, UI-Venus, and UI-Venus2 demonstrate applicability to models with different grounding capabilities and training histories. The extension to the Qwen3-VL-based UI-Venus and Qwen3.5-based UI-Venus2 also broadens the empirical evidence beyond Qwen2.5-VL-based models. For models exposing coordinate-token likelihoods and supporting the required visual reasoning, the framework offers a practical way to improve an existing inference pipeline.

  3. Broad and convincing empirical improvements, including on a strong contemporary GUI agent. The evaluation covers challenging professional interfaces, general mobile and web interfaces, and diverse step-level grounding tasks. The gains are substantial on ScreenSpot-Pro and remain positive on ScreenSpot-V2 even when baseline accuracy is already high.

Critical weaknesses
  1. Make the deployment implications of the accuracy–latency trade-off more explicit. The latency analysis is a valuable addition and resolves the initial concern about missing runtime measurements. Under the default setting, however, AutoFocus increases average runtime from 3.26 to 14.91 seconds per example, approximately a 4.6× increase. This is a reasonable trade-off for some accuracy-sensitive applications, but its suitability depends on the interaction setting. The paper would be more useful to practitioners if it highlighted both an accuracy-oriented configuration and a lower-latency configuration, with guidance on when each is appropriate.

  2. Further characterize the transferability of the uncertainty signal and refinement decisions. The cross-model evidence, including the new UI-Venus2 results, and the additional analyses described in the author response strengthen the use of perplexity as an uncertainty proxy. A useful next step would be to explain more clearly how its behavior changes across interface domains and failure types. In particular, errors caused by insufficient visual resolution may respond differently to refinement than errors caused by semantic confusion or confidently incorrect predictions.

  3. Consolidate implementation and comparison details for reproducibility. The appendix provides useful settings, including model-dependent exploration scales. The final version would benefit from a compact implementation summary specifying verifier and aggregator checkpoints, prompts, coordinate-token extraction, proposal filtering, and the procedure used to select hyperparameters.

Questions for authors
  1. Could the final version identify a recommended lower-latency configuration alongside the default setting?

  2. Building on the existing uncertainty and verifier analyses, could you summarize which error types benefit most from refinement and which remain difficult to recover from?

  3. Could you provide a consolidated description of the verifier and aggregator configurations and the hyperparameter-selection procedure, clarifying whether hyperparameters were selected on a separate validation split and whether reasoning models such as UI-Venus2 required model-specific adjustments?

Reviewer_iWLWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses small-target localization in high-resolution GUIs. AutoFocus visually verifies an initial prediction and, if verification fails, samples coordinate hypotheses. It maps the perplexities of x/y coordinate tokens to anisotropic Gaussian scales, generates global and local crops, and combines refined predictions through multi-image visual selection. The procedure requires no additional parameter updates and yields consistent improvements across several backbones on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G.

Critical strengths

S1: AutoFocus improves GUI grounding without additional training or parameter updates. Consistent gains across general-purpose and GUI-specialized VLMs on three benchmarks demonstrate the effectiveness of the refinement strategy across different model families and GUI settings.

Critical weaknesses

W1: AutoFocus incurs substantial inference overhead, increasing latency from 3.26s to 14.91s per example (4.6×). However, the comparison in Table 6 matches only the sampling budget (N=5), while AutoFocus additionally requires verification, multiple zoom-in inferences, and visual aggregation. This makes it difficult to assess whether the accuracy gains justify the additional computation. Comparisons under matched inference time or computational budgets would provide stronger evidence of its accuracy–efficiency trade-off.

W2: The evaluation focuses on single-step grounding accuracy across three benchmarks , without assessing performance on multi-step interactive GUI tasks. Since errors can accumulate and interfaces change during execution, improved grounding accuracy does not necessarily translate into higher end-to-end task success. Evaluating AutoFocus in interactive GUI tasks would help establish whether its grounding improvements yield practical benefits in real-world agent workflows.

Questions for authors

What is the specific benefit of axial uncertainty modeling? AutoFocus improves ScreenSpot-Pro accuracy from 26.8% to 35.4%, but using standard sequence-level PPL already reaches 34.1% (Table 4). Given this relatively minor 1.3% gap and the absence of other uncertainty baselines, the advantage of axial PPL remains unclear. Could you compare axial PPL with other alternatives such as sample-based spatial variance or predictive entropy under a fixed refinement pipeline and inference budget? This would help clarify the unique contribution of axial PPL.

Please also see the questions in the weaknesses.

Reviewer_P75EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AutoFocus, a training-free test-time refinement framework for GUI grounding. An initial prediction is checked through visual self-verification. For predictions that fail this check, the framework samples coordinate hypotheses and maps their axis-wise token perplexities to anisotropic Gaussian kernels. The resulting spatial field guides global and local crop proposals, followed by shape-aware crop adjustment, high-resolution grounding, and visual candidate selection. The paper reports improvements across several general-purpose and GUI-specialized backbones on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G, together with component ablations and a limited inference-cost analysis.

Critical strengths
  1. The paper addresses an important limitation of GUI grounding: small targets in high-resolution interfaces. The reported gains span multiple backbones and three benchmarks, including improvements on GUI-specialized models. This makes the approach potentially useful as an inference-time enhancement without additional training.

  2. A useful region-proposal design. Using axis-wise generation confidence to construct anisotropic, multi-scale proposals is an interesting alternative to fixed crop geometries. The ablations provide initial evidence that axial perplexity and shape-aware adjustment contribute to the complete pipeline.

Critical weaknesses
  1. The incremental contribution is not sufficiently isolated from prior refinement pipelines and additional computation. RegionFocus already includes visual self-verification, multiple cropped predictions, and visual candidate aggregation; UI-Zoomer already provides uncertainty-driven adaptive zooming. The main distinction therefore lies in the proposed spatial field and region construction. However, the experiments do not adequately isolate this distinction while holding the verifier, aggregator, and total inference budget fixed. In particular, matching the number of initial samples in Appendix A.4 does not match the additional verification, crop inference, and aggregation costs. This makes it difficult to attribute the reported gains specifically to the proposed uncertainty modeling.

  2. The spatial interpretation of perplexity and the effectiveness of the refinement gate remain insufficiently validated. The reported association between perplexity and prediction correctness does not establish that axis-wise perplexity is calibrated to axis-wise localization error, nor justify the linear mapping from perplexity to Gaussian scale. Moreover, the paper does not report per-backbone/per-benchmark gate activation rates or the gate’s ability to distinguish correct from incorrect predictions. Without these measurements, it remains unclear whether the method identifies spatial uncertainty reliably or mainly benefits from additional visual refinement.

Questions for authors
  1. Controlled comparison. With the backbone, verifier, aggregator, and total inference budget held fixed, how does the proposed region generator compare with RegionFocus-style fixed crops and sample-variance-based adaptive crops? Such a comparison would directly affect my assessment of the incremental contribution.

  2. Uncertainty validation. What evidence connects PPL _x and PPL_y to the corresponding localization residuals or empirical region coverage? How does the linear scale mapping compare with constant kernel scales and scales derived from sample dispersion?

  3. Gate behavior. Please report refinement activation rates, rejection rates conditional on initially correct/incorrect predictions, and correction/degradation rates after refinement for each evaluated backbone and benchmark. Is the average zoom count in Table 9 computed over all examples or only triggered examples?

  4. Experimental clarification. Please specify the predictor/verifier/aggregator checkpoints, prompts, and image budgets, and distinguish reproduced baselines from cited results. In particular, what explains the 27.5 versus 30.5 sampling-only results in Tables 4 and 6, and the UI-Venus baseline differences from the original reports?

Comments on the paper

1 comment
Area_Chair_SHKROpenReview ↗

Reviewers P75E & iWLW] Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers P75E & iWLW,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

Reviewer QBsy already responded and reconsider the rating of this submission.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.