- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes INTENTLOC, which reframes GUI grounding as an intent-level object localization problem rather than directly requiring VLMs to generate coordinates in text form. Also unlike prior such grounding approaches, e.g., GUI-Actor that directly score visual patches against a grounding token, INTENTLOC introduces a detector-style localization head that iteratively refines a VLM-derived grounding query. The key component is Crosshair Attention, where each decoder layer predicts an explicit 2D reference point (crosshair) and uses it to guide cross-attention through a spatial Gaussian bias. The predicted crosshairs are directly supervised with ground-truth locations, enabling a coarse-to-fine localization process. The authors also introduce Atypical-GUI-Bench, a benchmark designed to evaluate grounding robustness under non-canonical GUI observations. Experiments on multiple GUI grounding benchmarks demonstrate consistent improvements over prior methods, including GUI-Actor.
Critical strengths
1. Novel and intuitive exploration of detector-style modeling for GUI grounding.
Recent GUI grounding research has been dominated by text-based coordinate prediction methods, where scaling up training data and training VLMs to generate coordinate tokens appear to be promising solutions. However, the fundamental formulation of GUI grounding itself has received relatively limited investigation. This paper directly addresses this gap by exploring a detector-style formulation for GUI grounding, which is both novel and intuitive. Moreover, enabling iterative refinement within the model through detector-style decoder layers offers a promising alternative to existing multi-round cropping or zooming approaches.
2. Convincing empirical improvements and ablation studies.
The authors provide ablation studies that demonstrate the effectiveness of the proposed components. The method also achieves notable performance improvements over the previous GUI-Actor baseline, highlighting the potential of this alternative formulation for GUI grounding.
Critical weaknesses
1. Limited validation with stronger modern VLM backbones and alternative iterative baselines.
The main INTENTLOC models are built on Qwen2.5-VL, while newer VLMs such as Qwen3-VL already provide substantially stronger grounding capabilities. Although using the same backbone and training data as GUI-Actor ensures a fair comparison, evaluating INTENTLOC on stronger or state-of-the-art VLM backbones would help assess the scalability and generalizability of the proposed approach.
Additionally, since a major motivation is iterative coarse-to-fine localization, it would be valuable to compare against simpler iterative extensions of existing grounding models, such as applying GUI-Actor multiple times with a zoom-in/crop refinement strategy. Such comparisons, considering both grounding accuracy and inference efficiency, would help clarify the performance-efficiency trade-offs and whether the improvements stem from the proposed detector-style architecture specifically or from iterative localization more generally.
Questions for authors
N/A
5 comments
Response 1 to Weakness 2: Comparison with zoom-in strategy and analysis
W2. The source of the gains remains unclear without comparisons to zoom/crop baselines.
Response 1: External zoom helps hard cases but is less accurate and efficient than internal refinement.
We thank the reviewer for proposing this important baseline. IntentLoc’s coarse-to-fine process differs from external crop/zoom pipelines: its crosshairs are updated between decoder layers within one model forward pass. A crop/zoom pipeline instead invokes the grounder repeatedly, changes the visual input between rounds, and may lose the target after an incorrect early crop. This distinction explains the mechanism, but does not replace an empirical comparison; we therefore add the controlled GUI-Actor zoom/crop evaluation below.
We evaluate two crop-processing strategies: upscale/zoom, which enlarges each crop, and native/crop, which preserves its native resolution. Starting from the full-image prediction, each round centers a half-sized crop on the previous prediction, maps the new prediction back to the original image, and repeats for up to three rounds.
Table R14. Accuracy–efficiency comparison of iterative GUI-Actor and single-pass IntentLoc.
| Setting | ScreenSpot-Pro | ScreenSpot-v2 | OSWorld-G | UI-I2E-Bench | Forward passes | Cumulative latency (s/sample) | Cumulative visual tokens |
|---|---|---|---|---|---|---|---|
| GUI-Actor-7B | 44.6 | 91.8 | 42.2 | 68.2 | 1 | 0.98 | 4,402 |
| GUI-Actor-7B + 1 zoom round | 57.6 | 90.6 | 49.8 | 65.4 | 2 | 1.96 | 8,805 |
| GUI-Actor-7B + 2 zoom rounds | 59.5 | 88.1 | 51.2 | 63.7 | 3 | 2.95 | 13,207 |
| GUI-Actor-7B + 3 zoom rounds | 60.2 | 86.8 | 51.6 | 60.5 | 4 | 3.96 | 17,609 |
| GUI-Actor-7B + 1 native-crop round | 51.3 | 90.2 | 44.7 | 65.3 | 2 | 1.23 | 5,525 |
| GUI-Actor-7B + 2 native-crop rounds | 53.2 | 87.6 | 45.4 | 64.2 | 3 | 1.35 | 5,807 |
| GUI-Actor-7B + 3 native-crop rounds | 53.0 | 86.8 | 47.2 | 61.8 | 4 | 1.44 | 5,877 |
| IntentLoc-GUI-7B (3-layer, single pass) | 58.1 | 94.2 | 56.4 | 84.4 | 1 | 1.01 | 4,402 |
Whether zoom helps is influenced by two points: how many input patches the target occupies and whether cropping removes the target or its necessary context. ScreenSpot-Pro and OSWorld-G contain extremely small targets and leave substantial room for correcting baseline errors; zoom therefore repairs more predictions than it disrupts. In contrast, ScreenSpot-v2 contains larger targets and has an already strong single-pass baseline, leaving less room for improvement while introducing the risk of truncating a large target or discarding useful context. UI-I2E-Bench presents a related failure mode: its input fields are often wide and elongated, making them vulnerable to truncation (23.5% of input-field examples are disrupted, of which 89% involve truncation), while small toggle controls often depend on same-row labels that can be removed by cropping (18.6% disruption rate).
Even if we grant GUI-Actor an oracle choice of the best zoom round separately for each benchmark, it reaches a macro-average accuracy of only 67.93% with an average latency of 2.47 s. In comparison, single-pass IntentLoc achieves 73.27% with a latency of 1.01 s. These results show that external zoom can substantially help on small-target benchmarks, but its gains are benchmark-dependent and come with repeated-forward-pass costs; IntentLoc provides a stronger overall accuracy–efficiency trade-off through internal refinement.
Response 2 to Weakness 2: Comparison with zoom-in strategy and analysis
W2. The source of the gains remains unclear without comparisons to zoom/crop baselines
Response 2: Detector-style supervision and layer-wise refinement jointly drive the gains
We thank the reviewer for asking us to disentangle these two sources of improvement. The answer is both, and our results indicate that the proposed detector-style architecture accounts for the larger share of the improvement.
Appendix J isolates the contribution of layer-wise coarse-to-fine refinement by varying only the number of decoder layers. A one-layer decoder retains the detector-style architecture and spatial objective, but has no cross-layer refinement: it obtains 54.2/52.4/81.8 on ScreenSpot-Pro, OSWorld-G, and UI-I2E-Bench. Increasing the decoder to three layers improves the results to 58.1/56.4/84.4, corresponding to gains of +3.9/+4.0/+2.6 points. Two layers already improve over one layer (56.7/55.9/83.1), while four layers (56.3/56.1/84.4) provide no further gain, indicating that refinement is useful but saturates after a small number of stages.
As a more direct control that does not change decoder depth or capacity, Appendix J also keeps the decoder fixed at three layers but applies the crosshair only in the first layer. This variant obtains 56.4/55.0/82.7, compared with 58.1/56.4/84.4 when all three layers update the crosshair, yielding gains of +1.7/+1.4/+1.7 points. This fixed-depth comparison further supports the contribution of progressive layer-wise refinement.
Response 1,2 to Weakness 1 (newer backbone)
We thank Reviewer MXDU for recognizing the novelty and intuitiveness of our detector-style formulation and internal iterative refinement, as well as the convincing gains over GUI-Actor and supporting ablations; below, we respond with new experiments on stronger backbones and alternative coarse-to-fine baselines.
W1. stronger or state-of-the-art VLM backbones
Response 1: Newer backbones improve grounding accuracy
We thank the reviewer for this suggestion. We originally used Qwen2.5-VL primarily to enable a controlled comparison with GUI-Actor under the same backbone and training data. To directly evaluate whether the proposed architecture generalizes beyond this backbone, we have now instantiated IntentLoc with the newer Qwen3-VL-8B and Qwen3.5-4B backbones.
Table R11. IntentLoc accuracy with different VLM backbones.
Replacing Qwen2.5-VL-7B with Qwen3-VL-8B yields an average gain of 3.65 points across four benchmarks. Notably, Qwen3.5-4B achieves the best performance on all benchmarks despite having only half the backbone parameters of Qwen3-VL-8B, outperforming Qwen2.5-VL-7B and Qwen3-VL-8B by 5.13 and 1.48 points on average, respectively. These results demonstrate that IntentLoc generalizes effectively across VLM families and benefits from stronger visual-semantic representations.
IntentLoc built on the newer backbones remain competitve compared with contemporary works, e.g., IntentLoc-GUI-4B ranking second overall on ScreenSpot-Pro and OSWorld-G and first overall on MMBench-GUI-L2 and UI-I2E-Bench. It outperforms VISTA-4B by +0.7/+3.0 points on ScreenSpot-Pro/OSWorld-G, and surpasses DeskForge v1.1 by +15.9/+0.4/+8.5 points on ScreenSpot-Pro/OSWorld-G/MMBench-GUI-L2. We will further include Qwen3.5-9B backbone results in the revised paper.
Response 2: Newer backbones improve end-to-end OSWorld score
We also evaluate Qwen3.5-4B-based IntentLoc alongside the previous IntentLoc-GUI-7B under identical Opus 5.5 and Sonnet 5.5 planner settings, reporting both the OSWorld score and efficiency in Table R12.
Table R12. OSWorld performance and efficiency of IntentLoc with Qwen2.5-VL and Qwen3.5 backbones.
Relative to the planner-only baselines, IntentLoc-GUI-4B improves the OSWorld score by +3.44 points with Opus and +5.62 points with Sonnet, while also reducing planner-token usage and average task time. Compared with IntentLoc-GUI-7B, it raises the score from 78.87 to 80.28 with Opus and from 72.50 to 74.04 with Sonnet. This confirms that improvements from the newer grounding backbone translate into stronger end-to-end agent performance across different planners.