acceptodds
Under review as a conference paper at ICLR 2027

Reframing GUI Grounding as Intent-Level Object Detection

16 views

Abstract

Currently, Vision-Language Model (VLM)-based GUI agents formulate grounding as coordinate prediction, forcing the model to learn a mapping from spatial locations to discrete coordinate tokens without explicit spatial supervision. To address this limitation, we revisit GUI grounding from a spatial detection perspective and find that existing object detectors propose GUI-element-like regions but rarely rank the intended one first, i.e., they lack high-level intent understanding. We therefore propose IntentLoc, an intent-level object detection model that uses high-level semantic queries to equip a detector with intent understanding ability and introduces Crosshair Attention to progressively locate small GUI targets. Under the same backbone and training data, IntentLoc-GUI outperforms pure coordinate prediction baselines by 10.6 points, averaged with equal weight over four GUI grounding benchmarks and two model sizes. We further introduce Atypical-GUI-Bench, a manually curated benchmark for evaluating the robustness and generalization of GUI grounding models under atypical in-the-wild visual observations. Additionally, when integrated into an agentic framework with a VLM planner, our approach improves success on end-to-end GUI tasks. Code, all data (training data and benchmark), and models will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, object detection

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MXDUOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes INTENTLOC, which reframes GUI grounding as an intent-level object localization problem rather than directly requiring VLMs to generate coordinates in text form. Also unlike prior such grounding approaches, e.g., GUI-Actor that directly score visual patches against a grounding token, INTENTLOC introduces a detector-style localization head that iteratively refines a VLM-derived grounding query. The key component is Crosshair Attention, where each decoder layer predicts an explicit 2D reference point (crosshair) and uses it to guide cross-attention through a spatial Gaussian bias. The predicted crosshairs are directly supervised with ground-truth locations, enabling a coarse-to-fine localization process. The authors also introduce Atypical-GUI-Bench, a benchmark designed to evaluate grounding robustness under non-canonical GUI observations. Experiments on multiple GUI grounding benchmarks demonstrate consistent improvements over prior methods, including GUI-Actor.

Critical strengths

1. Novel and intuitive exploration of detector-style modeling for GUI grounding.

Recent GUI grounding research has been dominated by text-based coordinate prediction methods, where scaling up training data and training VLMs to generate coordinate tokens appear to be promising solutions. However, the fundamental formulation of GUI grounding itself has received relatively limited investigation. This paper directly addresses this gap by exploring a detector-style formulation for GUI grounding, which is both novel and intuitive. Moreover, enabling iterative refinement within the model through detector-style decoder layers offers a promising alternative to existing multi-round cropping or zooming approaches.

2. Convincing empirical improvements and ablation studies.

The authors provide ablation studies that demonstrate the effectiveness of the proposed components. The method also achieves notable performance improvements over the previous GUI-Actor baseline, highlighting the potential of this alternative formulation for GUI grounding.

Critical weaknesses

1. Limited validation with stronger modern VLM backbones and alternative iterative baselines.

The main INTENTLOC models are built on Qwen2.5-VL, while newer VLMs such as Qwen3-VL already provide substantially stronger grounding capabilities. Although using the same backbone and training data as GUI-Actor ensures a fair comparison, evaluating INTENTLOC on stronger or state-of-the-art VLM backbones would help assess the scalability and generalizability of the proposed approach.

Additionally, since a major motivation is iterative coarse-to-fine localization, it would be valuable to compare against simpler iterative extensions of existing grounding models, such as applying GUI-Actor multiple times with a zoom-in/crop refinement strategy. Such comparisons, considering both grounding accuracy and inference efficiency, would help clarify the performance-efficiency trade-offs and whether the improvements stem from the proposed detector-style architecture specifically or from iterative localization more generally.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response 1,2 to Weakness 1 (newer backbone)

We thank Reviewer MXDU for recognizing the novelty and intuitiveness of our detector-style formulation and internal iterative refinement, as well as the convincing gains over GUI-Actor and supporting ablations; below, we respond with new experiments on stronger backbones and alternative coarse-to-fine baselines.

W1. stronger or state-of-the-art VLM backbones

Response 1: Newer backbones improve grounding accuracy

We thank the reviewer for this suggestion. We originally used Qwen2.5-VL primarily to enable a controlled comparison with GUI-Actor under the same backbone and training data. To directly evaluate whether the proposed architecture generalizes beyond this backbone, we have now instantiated IntentLoc with the newer Qwen3-VL-8B and Qwen3.5-4B backbones.

Table R11. IntentLoc accuracy with different VLM backbones.

ModelBackboneRelease DateScreenSpot-ProOSWorld-GMMBench-GUI-L2UI-I2E-Bench
UI-Ins-7BQwen2.5-VL-7B2025-1052.2–83.181.1
GUI-Owl-7BQwen2.5-VL-7B2025-0854.955.980.5–
VISTA-8BQwen3-VL-8B2026-0665.862.486.8–
GroundGUI-8BQwen3-VL-8B2026-1063.965.486.8–
VISTA-4BQwen3.5-4B2026-0664.261.2––
DeskForge-Qwen3.5-4B (v1.1)Qwen3.5-4B2026-1049.063.880.1–
IntentLoc-GUI-7BQwen2.5-VL-7B–58.156.485.784.4
IntentLoc-GUI-8BQwen3-VL-8B–62.663.487.585.7
IntentLoc-GUI-4BQwen3.5-4B–64.964.288.687.4

Replacing Qwen2.5-VL-7B with Qwen3-VL-8B yields an average gain of 3.65 points across four benchmarks. Notably, Qwen3.5-4B achieves the best performance on all benchmarks despite having only half the backbone parameters of Qwen3-VL-8B, outperforming Qwen2.5-VL-7B and Qwen3-VL-8B by 5.13 and 1.48 points on average, respectively. These results demonstrate that IntentLoc generalizes effectively across VLM families and benefits from stronger visual-semantic representations.

IntentLoc built on the newer backbones remain competitve compared with contemporary works, e.g., IntentLoc-GUI-4B ranking second overall on ScreenSpot-Pro and OSWorld-G and first overall on MMBench-GUI-L2 and UI-I2E-Bench. It outperforms VISTA-4B by +0.7/+3.0 points on ScreenSpot-Pro/OSWorld-G, and surpasses DeskForge v1.1 by +15.9/+0.4/+8.5 points on ScreenSpot-Pro/OSWorld-G/MMBench-GUI-L2. We will further include Qwen3.5-9B backbone results in the revised paper.

Response 2: Newer backbones improve end-to-end OSWorld score

We also evaluate Qwen3.5-4B-based IntentLoc alongside the previous IntentLoc-GUI-7B under identical Opus 5.5 and Sonnet 5.5 planner settings, reporting both the OSWorld score and efficiency in Table R12.

Table R12. OSWorld performance and efficiency of IntentLoc with Qwen2.5-VL and Qwen3.5 backbones.

SystemOSWorld scorePlanner tokensPlanner latencyGrounding latencyAvg. time/task
Opus 5.576.8473.5K65 s–3.9 min
Opus 5.5 + IntentLoc-GUI-7B (backbone: Qwen2.5-VL-7B)78.87 (↑2.03)70.4K (↓3.1K)58 s (↓7 s)7 s3.7 min (↓0.2 min)
Opus 5.5 + IntentLoc-GUI-4B (backbone: Qwen3.5-4B)80.28 (↑3.44)69.5K (↓4.0K)56 s (↓9 s)5 s3.6 min (↓0.3 min)
Sonnet 5.568.42114.6K116 s–4.8 min
Sonnet 5.5 + IntentLoc-GUI-7B (backbone: Qwen2.5-VL-7B)72.50 (↑4.08)91.1K (↓23.5K)68 s (↓48 s)6 s4.2 min (↓0.6 min)
Sonnet 5.5 + IntentLoc-GUI-4B (backbone: Qwen3.5-4B)74.04 (↑5.62)93.8K (↓20.8K)69 s (↓47 s)5 s4.2 min (↓0.6 min)

Relative to the planner-only baselines, IntentLoc-GUI-4B improves the OSWorld score by +3.44 points with Opus and +5.62 points with Sonnet, while also reducing planner-token usage and average task time. Compared with IntentLoc-GUI-7B, it raises the score from 78.87 to 80.28 with Opus and from 72.50 to 74.04 with Sonnet. This confirms that improvements from the newer grounding backbone translate into stronger end-to-end agent performance across different planners.

AuthorsOpenReview ↗

Response 1 to Weakness 2: Comparison with zoom-in strategy and analysis

W2. The source of the gains remains unclear without comparisons to zoom/crop baselines.

Response 1: External zoom helps hard cases but is less accurate and efficient than internal refinement.

We thank the reviewer for proposing this important baseline. IntentLoc’s coarse-to-fine process differs from external crop/zoom pipelines: its crosshairs are updated between decoder layers within one model forward pass. A crop/zoom pipeline instead invokes the grounder repeatedly, changes the visual input between rounds, and may lose the target after an incorrect early crop. This distinction explains the mechanism, but does not replace an empirical comparison; we therefore add the controlled GUI-Actor zoom/crop evaluation below.

We evaluate two crop-processing strategies: upscale/zoom, which enlarges each crop, and native/crop, which preserves its native resolution. Starting from the full-image prediction, each round centers a half-sized crop on the previous prediction, maps the new prediction back to the original image, and repeats for up to three rounds.

Table R14. Accuracy–efficiency comparison of iterative GUI-Actor and single-pass IntentLoc.

SettingScreenSpot-ProScreenSpot-v2OSWorld-GUI-I2E-BenchForward passesCumulative latency (s/sample)Cumulative visual tokens
GUI-Actor-7B44.691.842.268.210.984,402
GUI-Actor-7B + 1 zoom round57.690.649.865.421.968,805
GUI-Actor-7B + 2 zoom rounds59.588.151.263.732.9513,207
GUI-Actor-7B + 3 zoom rounds60.286.851.660.543.9617,609
GUI-Actor-7B + 1 native-crop round51.390.244.765.321.235,525
GUI-Actor-7B + 2 native-crop rounds53.287.645.464.231.355,807
GUI-Actor-7B + 3 native-crop rounds53.086.847.261.841.445,877
IntentLoc-GUI-7B (3-layer, single pass)58.194.256.484.411.014,402

Whether zoom helps is influenced by two points: how many input patches the target occupies and whether cropping removes the target or its necessary context. ScreenSpot-Pro and OSWorld-G contain extremely small targets and leave substantial room for correcting baseline errors; zoom therefore repairs more predictions than it disrupts. In contrast, ScreenSpot-v2 contains larger targets and has an already strong single-pass baseline, leaving less room for improvement while introducing the risk of truncating a large target or discarding useful context. UI-I2E-Bench presents a related failure mode: its input fields are often wide and elongated, making them vulnerable to truncation (23.5% of input-field examples are disrupted, of which 89% involve truncation), while small toggle controls often depend on same-row labels that can be removed by cropping (18.6% disruption rate).

Even if we grant GUI-Actor an oracle choice of the best zoom round separately for each benchmark, it reaches a macro-average accuracy of only 67.93% with an average latency of 2.47 s. In comparison, single-pass IntentLoc achieves 73.27% with a latency of 1.01 s. These results show that external zoom can substantially help on small-target benchmarks, but its gains are benchmark-dependent and come with repeated-forward-pass costs; IntentLoc provides a stronger overall accuracy–efficiency trade-off through internal refinement.

AuthorsOpenReview ↗

Response 2 to Weakness 2: Comparison with zoom-in strategy and analysis

W2. The source of the gains remains unclear without comparisons to zoom/crop baselines

Response 2: Detector-style supervision and layer-wise refinement jointly drive the gains

We thank the reviewer for asking us to disentangle these two sources of improvement. The answer is both, and our results indicate that the proposed detector-style architecture accounts for the larger share of the improvement.

Appendix J isolates the contribution of layer-wise coarse-to-fine refinement by varying only the number of decoder layers. A one-layer decoder retains the detector-style architecture and spatial objective, but has no cross-layer refinement: it obtains 54.2/52.4/81.8 on ScreenSpot-Pro, OSWorld-G, and UI-I2E-Bench. Increasing the decoder to three layers improves the results to 58.1/56.4/84.4, corresponding to gains of +3.9/+4.0/+2.6 points. Two layers already improve over one layer (56.7/55.9/83.1), while four layers (56.3/56.1/84.4) provide no further gain, indicating that refinement is useful but saturates after a small number of stages.

As a more direct control that does not change decoder depth or capacity, Appendix J also keeps the decoder fixed at three layers but applies the crosshair only in the first layer. This variant obtains 56.4/55.0/82.7, compared with 58.1/56.4/84.4 when all three layers update the crosshair, yielding gains of +1.7/+1.4/+1.7 points. This fixed-depth comparison further supports the contribution of progressive layer-wise refinement.

Reviewer_aJTKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates intent-level object detection using a pretrained visual language model (VLM) as a backbone, providing an alternative pathway towards a stronger GUI agent with better semantic alignment and more precise localization capability. By introducing both patch-loss and crosshair loss, the authors claim their method can progressively guide the model to locate small GUI targets in the input image. Compared with existing baselines, the proposed method, IntentLoc, outperforms others by 10.6 points on average, suggesting a significant improvement. On Atypical-GUI-Bench, which is a set of atypical in-the-wild GUI tasks, IntentLoc showcases a more surprising generalizability than other models.

Critical strengths
  • This work introduces a new decoder with crosshair attention, layer-wise supervision for GUI grounding, and directed cross-attention readout to enable a finer-grained learning signal for the intent-level object detection objective.
  • The paper shows suffcient experiment results, across different model sizes, benchmarks and an atypical environment, to verify its capability and generalizability against other existing methods.
  • The authors also provide a new and manually curated benchmark, Atypical-GUI-Bench, focusing on evaluating the robustness and generalization of GUI agents under atypical visual inputs.
Critical weaknesses
  • The authors claim that semantic ranking is a primary bottleneck of existing GUI grounding models, e.g., GroundingDINO; however, the experimental results in Appendix D show that the recall of top-10 candidates is only 9.45%, which suggests that insufficient recall seems to be a bigger challenge.
  • The data construction process of the proposed new benchmark Atypical-GUI-Bench follows the practice of synthesizing instructions given a screenshot (Appendix E & F). This target-aware instruction construction is potentially vulnerable since the model could still learn instruction-to-element matching rather than explicit intention deduction.
Questions for authors
  • Can the authors provide more empirical results to show to what extent the proposed IntentLoc addresses the semantic ranking challenge that previous methods were considered to face?
  • Intent-level object detection is the core contribution of this work. Can the authors provide more direct evidence to show that the model is stronger at intention reasoning rather than simple instruction-target mapping?
4 comments
AuthorsOpenReview ↗

Response to Weakness 1 (Is low candidate recall a larger problem than semantic ranking?)

We thank Reviewer aJTK for recognizing our Crosshair Attention decoder and fine-grained supervision, the breadth of our evaluation, and the value of the manually curated Atypical-GUI-Bench; below, we address the reviewer’s questions about candidate recall, semantic ranking, and intent reasoning point by point.

W1. Is low candidate recall a larger problem than semantic ranking?

Response: Recall is severe on ScreenSpot-Pro, while semantic ranking remains a distinct detector bottleneck

We thank the reviewer for raising this important concern about candidate recall. Appendix D does show that the recall of top-10 candidates is only 9.45% on ScreenSpot-Pro. This is because ScreenSpot-Pro is an exceptionally challenging benchmark: its most common screen resolution is 2560×1440, while the average target occupies only 0.07% of the image. Therefore, achieving around 10% on this benchmark already indicates non-trivial GUI grounding ability. For context, the official leaderboard reports only 1.1% accuracy for the earlier 9.6B SeeClick and 0.8% for GPT-4o, while AriaUI (MoE, 3.9B active parameters) achieves 11.3%, comparable to GroundingDINO's Hit@10 trained on 1M examples.

More importantly, the low ScreenSpot-Pro number does not reflect a general inability to recall GUI elements. As shown in Figure 2, the same 1M-data trained GroundingDINO achieves Hit@10 scores of 66.00% on the in-distribution validation set, 64.31% on ScreenSpot, and 63.21% on ScreenSpot-v2. The sharp drop on ScreenSpot-Pro mainly reflects its unusually dense, high-resolution professional interfaces. We will clarify this benchmark difficulty and avoid interpreting the 9.45% result in isolation.

We nevertheless agree that, on ScreenSpot-Pro specifically, insufficient target recall is the more immediate bottleneck: if the intended target is absent from the Top-10 candidates, reranking cannot recover it. Our claim is comparative rather than exclusive. Target localization is difficult for all model families on this benchmark, as reflected by the similarly low accuracy of LLM/VLM-based grounders. What our analysis emphasizes is that conventional detectors have substantially weaker semantic selection than LLM-based grounders: their large Hit@1-to-Hit@10 gap and the strong gains from OCR reranking show that many valid candidates are poorly ranked according to the instruction. We will therefore revise “the primary bottleneck” to “a major bottleneck for conventional detectors,” while explicitly acknowledging that recall is also a major challenge on ScreenSpot-Pro.

AuthorsOpenReview ↗

Response 1,2,3 to Weakness 2: Target-aware instruction construction and direct evidence of intent reasoning

W2. Target-aware instruction construction and direct evidence of intent reasoning

Response 1: Atypical-GUI-Bench serves as visual robustness evaluation benchmark rather than intent reasoning training/testing data.

We thank the reviewer for highlighting this "missing intent reasoning" problem in the data construction. We agree that constructing an instruction after selecting a screenshot and target does not by itself incorporate intention reasoning. A model could exploit explicit target names or superficial lexical correspondence. But Atypical-GUI-Bench was designed primarily to test robustness under non-canonical visual observations, not to serve as a clean causal test of intent reasoning. We will clarify this scope and avoid using its construction procedure alone as evidence for intent understanding.

Importantly, Atypical-GUI-Bench is used only for evaluation and is never included in training. Its target-aware construction therefore cannot introduce a training shortcut or cause the model to learn instruction-to-element mappings; the reviewer’s concern pertains to what the benchmark measures, rather than to how our model is trained.

Response 2: The addtional 0.25M training corpus covers all three instruction abstraction levels.

We thank the reviewer for motivating us to make the instruction composition explicit. We acknowledge that the original manuscript omitted or mistook an important detail about our annotation process. Our training corpus is not limited to target-span descriptions; it contains all three instruction levels defined in Appendix C: target-span, single-step intent-level, and task-level prompts. During annotation of the additional 0.25M examples, we explicitly asked annotators to provide task-level prompts for a portion of the data, rather than always describing the target element directly. We subsequently audited this corpus, in which each screenshot is paired with one supervised interaction target. The approximate distribution is:

Table R6. Distribution and examples of the three instruction levels in our training corpus.

Instruction levelApprox. proportionRepresentative training prompts
Single-step intent-level35–40%“Switch to the stopwatch to start timing”; “Mute the currently playing video”; “Skip ahead in the current episode by 10 seconds”; “Switch the app to night mode to reduce eye strain”; “Change the map view to satellite imagery.”
Target-span level30–35%“A white circular button displaying the number ‘65’”; “the ECG-shaped icon in the center of the bottom navigation bar”; “Click the blue plus button next to the layout-grid panel.”
Task-level≈30%“Book a standard room for a three-night stay starting on August 10”; “Set an alarm that rings once at 11 a.m. tomorrow”; “Create a new account to participate in the electronics project”; “Change the length, width, and quantity of the item in row 6, and confirm.”

These proportions are approximate because surface-form heuristics can misclassify implicit prompts. Nevertheless, an additional manual audit confirms that intent- and task-oriented instructions constitute the clear majority of this subset. In contrast, our audit indicates that more than 95% of GUI-Actor data consists of direct element descriptions.

For task-level examples, annotators describe a multi-step objective, while the annotation identifies the next interaction target under the current screenshot. Thus, these deliberately collected samples expose the model to high-level task semantics while remaining compatible with single-target grounding supervision. Consequently, the model is trained not only to match explicit element names, but also to infer the actionable control from an intended operation or a broader task objective. This three-level training mixture directly reduces the risk that IntentLoc merely learns superficial instruction-to-element matching.

Response 3: Task-level benchmarks provide direct evidence beyond Atypical-GUI-Bench.

We thank the reviewer for requesting evidence beyond Atypical-GUI-Bench. Since Atypical-GUI-Bench primarily evaluates single-step intent-level prompts, we additionally evaluate IntentLoc with more abstract task-level prompts on Mind2Web and AndroidControl. Please see our detailed response to Question 2 below.

AuthorsOpenReview ↗

Response to Question 1: IntentLoc addresses the semantic ranking challenge

Q1. To what extent the proposed IntentLoc addresses the semantic ranking challenge

Response: Top-k saturation and ineffective reranking show that IntentLoc resolves semantic ranking

We thank the reviewer for asking us to quantify this claim directly. We have added new results showing that IntentLoc substantially resolves the semantic-ranking bottleneck. To recap, our pre-analysis defines semantic-ranking failure as a model localizing plausible GUI candidates but failing to rank the instruction-specified target first. For IntentLoc, Top-k accuracy saturates rapidly after Top-1, and an external semantic reranker provides almost no additional gain. We verify this conclusion through two complementary diagnostics below.

First, we provide a new Top-k decomposition on the full ScreenSpot-Pro test set (1,581 examples). We extract connected components from IntentLoc’s predicted patch distribution, rank them by their component scores, and evaluate whether any of the Top-k component centroids falls inside the ground-truth box.

Table R7. Top-k accuracy of IntentLoc on ScreenSpot-Pro.

MetricTop@1Top@3Top@5Top@10
Accuracy58.1362.1862.3762.49

Top-k accuracy nearly saturates at k=3: increasing k from 1 to 10 improves overall accuracy by only 4.36 points (58.13→62.49), and Top@3 to Top@10 adds only 0.31 points. Equivalently, IntentLoc’s Top-1 prediction already recovers 93.0% of all examples that could be solved by its Top-10 candidates (58.13/62.49). This is qualitatively different from GroundingDINO, whose large Hit@1-to-Hit@10 gap indicates that the correct target is frequently generated but ranked too low.

We further apply the same OCR-based semantic reranker used in our GroundingDINO analysis. Its parameters are selected using two-fold cross-validation to avoid tuning on the evaluated examples.

Table R8. Effect of OCR semantic reranking on IntentLoc.

SettingTop@1
IntentLoc (original)58.13
+ OCR reranker59.14 (↑1.01)
Top@10 oracle upper bound62.49

OCR reranking changes the ordering for only 16 of 1,581 examples and improves the top@1 accuracy by merely 1.01 points, or approximately 16 examples. In contrast, the same reranker improves GroundingDINO-Swin-T on ScreenSpot-Pro from 1.84 to 12.28 at Hit@1 and MM-GroundingDINO-Swin-L from 2.59 to 13.13. Thus, semantic reranking is highly effective for conventional detectors but provides almost no benefit for IntentLoc.

Together, these results directly quantify the extent to which IntentLoc addresses semantic ranking. Its VLM-derived intent query already places the correct localized component first in nearly all cases where that component is available among the Top-10. Consequently, the remaining errors are dominated by candidate localization/recall rather than by semantic ordering among localized candidates. We therefore conclude that IntentLoc substantially resolves the semantic-ranking bottleneck observed in GroundingDINO.

AuthorsOpenReview ↗

Response to Question 2 (More evidence of our method's ability in intent reasoning)

Q2. More direct evidence of our method's ability in intent reasoning

Response: Task-level Mind2Web and AndroidControl results demonstrate intent reasoning

We thank the reviewer for asking for direct evidence that IntentLoc performs intent reasoning rather than merely matching an instruction to an explicitly named element. Appendix C distinguishes three levels of abstraction: (1) task-level instructions describe a high-level, often multi-step goal (e.g., “Order a cup of milk tea”); (2) intent-level instructions specify a single operation on the current screen without necessarily naming its target (e.g., “Run compile in VSCode”); and (3) target-span-level instructions directly identify the visual element. Our existing ScreenSpot and ScreenSpot-Pro results primarily evaluate the second level. We now add evaluations at the more abstract task level, where the target cannot be recovered through direct lexical matching alone.

Mind2Web: task-level prompt versus element-level oracle. Mind2Web provides a high-level task description and the interaction target at each step. To create a controlled element-level upper bound, we construct an oracle prompt from the ground-truth step_repr, which directly exposes the target element’s HTML tag, visible text, and operation, for example:

  • [button] Change Location → CLICK
  • [searchbox] Search by city... → TYPE: New York
  • [option] New York, NY → CLICK

Because this element-level prompt is derived from the ground-truth annotation, it is not available in a realistic deployment; it serves only as an oracle target-span condition. The task-level condition instead requires the model to infer the next actionable element from the high-level goal and current interface state. None of the evaluated models was trained or fine-tuned on the Mind2Web training split; all results in Table R9 are obtained by direct zero-shot evaluation.

Table R9. Element-grounding accuracy on Mind2Web under task-level and oracle element-level prompts.

Model and promptCross-task (2,080)Cross-website (1,371)Cross-domain (5,817)Overall (9,268)
GUI-Actor-7B, task-level28.4630.1233.3531.78
UI-Ins-7B, task-level34.6235.5938.5837.25
IntentLoc-GUI-7B, task-level60.1457.1155.6856.89
IntentLoc-GUI-7B, element-level oracle72.9869.9578.0875.73

Under the realistic task-level prompt, IntentLoc reaches 56.89% overall, outperforming GUI-Actor-7B by 25.11 points and UI-Ins-7B by 19.64 points. The improvement is consistent across cross-task, cross-website, and cross-domain splits. Although the element-level oracle is higher at 75.73%, that gap is expected because the oracle directly reveals the target element. Crucially, IntentLoc retains strong grounding accuracy when this target identity is removed and must instead be inferred from the task. This controlled prompt comparison on the same underlying targets provides direct evidence that the model does more than instruction–target string matching.

AndroidControl: high-level goal versus low-level step instruction. AndroidControl provides two evaluation modes: AC-Low supplies a low-level step instruction, whereas AC-High supplies only the high-level goal.

Table R10. Grounding accuracy on AndroidControl.

ModelAC-Low (low-level prompt)AC-High (high-level prompt)
GUI-Actor-7B78.3155.25
UI-Ins-7B79.1153.93
IntentLoc-GUI-7B79.5257.79

IntentLoc outperforms both GUI-Actor and UI-Ins under both instruction granularities. Relative to GUI-Actor, it improves AC-Low and AC-High by +1.21 and +2.54 points, respectively; relative to UI-Ins, the corresponding gains are +0.41 and +3.86 points. For both baselines, the advantage is larger under the high-level setting. Together with the Mind2Web results, this shows that IntentLoc’s advantage becomes more pronounced when the prompt provides an abstract goal rather than the explicit identity of the target element. Combined with our three-level training mixture described above, these results directly support IntentLoc’s intent- and task-reasoning capability rather than simple instruction–element matching.

Reviewer_SJGkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors point out several problems with how current VLM-based grounding models output 2D coordinates as text tokens: the model has to implicitly learn a mapping from spatial positions to discrete tokens, there is no explicit spatial supervision so training is inefficient, and the same target turns into a different token sequence whenever the screenshot is rescaled. To address this, they propose IntentLoc, which extracts a semantic query from the VLM and uses Crosshair Attention to progressively narrow in on small targets. The method is evaluated mainly on GUI grounding benchmarks and validated with 3B and 7B models.

Critical strengths
  1. The improvements from the proposed method are consistent across benchmarks and model scales, and the ablation study does a good job of isolating what each component contributes.
  2. It is interesting that the paper questions the common practice of having a VLM directly predict 2D coordinates as text tokens. Most recent work simply takes this for granted, so it is good to see someone ask whether it is actually the right design.
Critical weaknesses
  1. I am not fully convinced the proposed framework would make its way into a real GUI agent. Most deployed agents are split into a planner and an executor, and a capable planner can already take over much of what this paper asks the grounder to do. The 3B/7B grounders used here are fairly small models that sit somewhat below the current SOTA. Against that kind of baseline, improving grounding accuracy is less surprising. In practice, I suspect most systems would still pair an LLM-based planner with a moderately sized grounder, and hand the semantic ranking problem from Section 3 to the planner, which is likely to do it much better. I did look at Table 3, but it leaves me with questions. A single not yet strong planner is used per benchmark, and when UI-Ins-7B or GUI-Actor-7B is combined with the planner, the gain over the planner alone is close to zero. That suggests either that these grounders are simply inaccurate, or that they have an exposure-bias problem with respect to the kind of instructions a planner emits. Either way, it makes it hard to read the agentic results as evidence that the proposed approach is what a real system needs.

  2. Section 3 also leaves me with questions. The analysis is done only on a fine-tuned GroundingDINO; I would like to know whether larger, more recent GUI grounders actually suffer from the same semantic-ranking problem.I think this connects to what we see when the method is scaled from 3B to 7B without the additional training data: the margin over the coordinate-prediction baseline drops, and the 7B model falls behind several contemporary 7B grounders.

Taken together, I am not sure the trade-off works out in a real system: while the planner’s API cost is being paid, the proposed method adds its own computation, and in return we get semantic ranking that can still be unreliable. For these reasons, the paper currently reads to me more as publication-oriented research than as an approach a production GUI agent would adopt.

Questions for authors

Please see the Weaknesses section.

7 comments
AuthorsOpenReview ↗

Response 1 to Weakness 1: practicality in a real GUI agent and the advantage of "planner + IntentLoc" method

We thank Reviewer SJGk for recognizing our consistent gains across benchmarks and model scales, the effectiveness of our component-wise ablations, and the importance of re-examining coordinate-token prediction; below, we respond point by point to the reviewer’s concerns about practical deployment and model scaling.

W1 Practicality in a real GUI agent

Response 1: “Detect all elements first, then let a powerful planner rank them” is not an accurate way.

We thank the reviewer for raising this important practical question. We respectfully disagree that semantic target selection can simply be delegated to the planner after a detector proposes all GUI elements. A natural implementation of this idea is the Set-of-Mark (SoM) paradigm: a detector-based UI parser such as OmniParser V2 (whose element detector is YOLO-based) first detects the candidate elements and overlays their boxes and identifiers, after which a frontier LLM/VLM selects the intended identifier. However, an independent practical study, Why Your AI Agent Keeps Misclicking? A Practical Guide to GUI Grounding for Frontier Models, reports that this decomposition performs poorly on dense professional GUIs. Specifically, the authors evaluated Claude-Sonnet-4.5, Seed1.8, and Kimi-K2.5 on the refined OSWorld-G benchmark, utilizing SoM extracted via OmniParser V2. Despite supplying the original unmarked screenshot alongside the SoM visualization to reduce occlusion, the resulting accuracies were only 13.7, 20.0, and 18.6, respectively for Claude-Sonnet-4.5, Seed1.8, and Kimi-K2.5 on the refined OSWorld-G benchmark; none reached 30%. The study attributes this failure to dense and overlapping UI elements, noting that “the visual noise is heavy” and that the resulting ambiguity impedes the reasoning model’s ability to select the correct identifier.

This evidence shows that “detect all elements first, then let a powerful planner rank them” is not a sufficient replacement for a specialized grounder: the large candidate set and box/ID overlays themselves introduce substantial visual noise, while the planner still faces a difficult fine-grained correspondence problem. IntentLoc instead performs instruction-conditioned dense localization internally, without rendering hundreds of detector proposals into the planner’s visual context. The planner remains responsible for task decomposition and action selection, whereas IntentLoc resolves the complementary problem of mapping each subgoal to a precise visual target.

References:

  1. Liangyu Chen, Hanzhang Zhou, Quyu Kong, Xu Zhang, Wenxuan Wang, Qin Jin, and Yue Wang. (2026). Why Your AI Agent Keeps Misclicking? A Practical Grounding Guide for Frontier Models. Blog Post. https://galvanized-jump-79a.notion.site/Why-your-AI-Agent-keeps-misclicking-A-Practical-Guide-to-GUI-Grounding-for-Frontier-Models-32630d140ad8808e895de98994dddb93
AuthorsOpenReview ↗

Response 2 to Weakness 1: practicality in a real GUI agent and the advantage of "planner + IntentLoc" method

W1 Practicality in a real GUI agent

Response 2: Executor gains depend on relative grounding quality, not exposure bias alone. Our Planner + IntentLoc Method still has advantages in accuracy and efficiency with stronger LLM planner.

We agree that an executor (i.e., a grounding model) is useful only when its grounding ability exceeds that already available to the planner. However, our results do not support exposure bias as the sole explanation for the small gains of UI-Ins and GUI-Actor. In the main paper and appendix, all three executors improve two different planners on OSWorld benchmark:

Table R1. OSWorld scores with the planners evaluated in the main paper and Appendix I.

SystemOSWorld score (Δ vs. planner only)
Qwen3-VL-235B-A22B-Thinking38.1
Qwen3-VL-235B-A22B-Thinking + UI-Ins-7B40.7 (↑2.6)
Qwen3-VL-235B-A22B-Thinking + GUI-Actor-7B38.7 (↑0.6)
Qwen3-VL-235B-A22B-Thinking + IntentLoc-GUI-7B44.2 (↑6.1)
Claude-Haiku-4.550.7
Claude-Haiku-4.5 + UI-Ins-7B56.9 (↑6.2)
Claude-Haiku-4.5 + GUI-Actor-7B56.3 (↑5.6)
Claude-Haiku-4.5 + IntentLoc-GUI-7B58.3 (↑7.6)

With Qwen3-VL, UI-Ins, GUI-Actor, and IntentLoc improve the OSWorld score by +2.6, +0.6, and +6.1 points, respectively. With Claude-Haiku-4.5, the corresponding gains are +6.2, +5.6, and +7.6 points. The gains across two planner families indicate that these executors can process planner-generated instructions; therefore, a systematic instruction-distribution mismatch is unlikely to be the only cause of their performance.

To test whether this conclusion holds with stronger, more recent planners, we additionally evaluate Hy4, Claude Opus 5.5, and Claude Sonnet 5.5 on OSWorld:

Evaluation setting. Within each planner group in this supplementary experiment, all variants use exactly the same planner API source and configuration as the corresponding planner-only baseline. The grounding models are served on 8 NVIDIA A100 GPUs.

Table R2. Supplementary OSWorld results with stronger recent planners. Planner and grounding latencies are average cumulative times per task, summed over all calls within that task. Avg. time/task is not simply the sum of planner and grounding latency but also includes environment interaction.

SystemOSWorld scorePlanner tokensPlanner latencyGrounding latencyAvg. time/task
Hy472.4788.1K7.0 min–10.4 min
Hy4 + UI-Ins-7B69.02 (↓3.45)89.2K (↑1.1K)7.5 min (↑0.5 min)9 s14.2 min (↑3.8 min)
Hy4 + GUI-Actor-7B73.49 (↑1.02)89.7K (↑1.6K)8.3 min (↑1.3 min)9 s12.0 min (↑1.6 min)
Hy4 + IntentLoc-GUI-7B74.99 (↑2.52)88.3K (↑0.2K)6.7 min (↓0.3 min)7 s10.2 min (↓0.2 min)
Opus 5.576.8473.5K65 s–3.9 min
Opus 5.5 + IntentLoc-GUI-7B78.87 (↑2.03)70.4K (↓3.1K)58 s (↓7 s)7 s3.7 min (↓0.2 min)
Sonnet 5.568.42114.6K116 s–4.8 min
Sonnet 5.5 + IntentLoc-GUI-7B72.50 (↑4.08)91.1K (↓23.5K)68 s (↓48 s)6 s4.2 min (↓0.6 min)

These results lead to three observations.

  • First, as planners become stronger, an older executor may cease to help: UI-Ins improves the weaker Qwen3-VL and Claude-Haiku planners but reduces Hy4 from 72.47 to 69.02. This pattern is better explained by the executor’s relative grounding accuracy than by exposure bias alone. Delegation helps only when the executor is more reliable at grounding than the planner itself.

  • Second, planner–executor design remains effective when paired with a sufficiently strong grounder. Although UI-Ins and GUI-Actor provide little or negative benefit with Hy4, our IntentLoc-GUI-7B model consistently improves all three recent planners: +2.52 points with Hy4, +2.03 with Opus 5.5, and +4.08 with Sonnet 5.5, for an average gain of 2.88 percentage points. This shows that the planner–executor design remains effective when paired with a sufficiently strong grounder.

  • Third, a specialized grounder can also reduce planner-side computation. Our execution traces show that planner-only systems often use iterative visual reasoning and zoom-in operations to compensate for grounding uncertainty. Delegating localization to IntentLoc avoids much of this process. For example, adding IntentLoc reduces Sonnet 5.5’s planner-token usage from 114.6K to 91.1K and average task time from 4.8 to 4.2 minutes, while improving the score by 4.08 points. Opus 5.5 exhibits the same trend, with fewer planner tokens and lower average task time. The summed grounding latency itself is modest: 7/7/6 seconds with Hy4/Opus 5.5/Sonnet 5.5, respectively. Thus, IntentLoc offers not only higher OSWorld scores but also a favorable accuracy–efficiency trade-off for strong planners.

AuthorsOpenReview ↗

Response 3 to Weakness 1: practicality in a real GUI agent and the advantage of "planner + IntentLoc" method

W1 Practicality in a real GUI agent

Response 3: Newer backbones allow IntentLoc to keep pace with stronger planners.

As planners improve, the specialized grounder can likewise be updated with newer VLM backbones and training data. To test this co-evolution directly, we replace the Qwen2.5-VL-7B backbone of IntentLoc-GUI-7B with Qwen3.5-4B and evaluate both versions with Opus 5.5 and Sonnet 5.5 under the same OSWorld setting.

Table R3. OSWorld performance and efficiency with updated IntentLoc backbones. Arrows and parenthesized values denote changes relative to the corresponding planner-only row. Planner and grounding latencies are average cumulative times per task, summed over all calls within that task.

SystemOSWorld scorePlanner tokensPlanner latencyGrounding latencyAvg. time/task
Opus 5.576.8473.5K65 s–3.9 min
Opus 5.5 + IntentLoc-GUI-7B (backbone: Qwen2.5-VL-7B)78.87 (↑2.03)70.4K (↓3.1K)58 s (↓7 s)7 s3.7 min (↓0.2 min)
Opus 5.5 + IntentLoc-GUI-4B (backbone: Qwen3.5-4B)80.28 (↑3.44)69.5K (↓4.0K)56 s (↓9 s)5 s3.6 min (↓0.3 min)
Sonnet 5.568.42114.6K116 s–4.8 min
Sonnet 5.5 + IntentLoc-GUI-7B72.50 (↑4.08)91.1K (↓23.5K)68 s (↓48 s)6 s4.2 min (↓0.6 min)
Sonnet 5.5 + IntentLoc-GUI-4B (backbone: Qwen3.5-4B)74.04 (↑5.62)93.8K (↓20.8K)69 s (↓47 s)5 s4.2 min (↓0.6 min)

The updated grounder consistently improves both planners. With Opus 5.5, IntentLoc-GUI-4B raises the score by 3.44 points, reduces planner tokens by 4.0K, and lowers average task time by 0.3 minutes. With Sonnet 5.5, it raises the score by 5.62 points, reduces planner tokens by 20.8K, and lowers average task time by 0.6 minutes. Compared with the earlier IntentLoc-GUI-7B executor, the Qwen3.5-4B version further improves the score by +1.41 points with Opus and +1.54 points with Sonnet, while reducing grounding latency from 7 to 5 seconds and 6 to 5 seconds, respectively. These results demonstrate that the planner–grounder architecture does not become obsolete as planners advance: the grounding component can co-evolve through newer backbones and data, and the updated IntentLoc continues to provide larger end-to-end gains.

These results also suggest a lower-cost deployment option. Sonnet 5.5 + IntentLoc-GUI-4B achieves a score of 74.04, only 2.80 points below Opus 5.5 alone at 76.84. Because Sonnet has a lower per-token API price than Opus, replacing Opus with Sonnet plus IntentLoc can substantially reduce API costs while preserving most of the task performance. This trade-off is particularly valuable for token-intensive computer-use workloads.

P.S. More evaluation results for IntentLoc with newer and stronger backbones can be found in Response 3 to Weakness 2 below.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Rebuttal Summary

We sincerely thank all reviewers for their careful reading, constructive criticism, and concrete suggestions. We are also grateful for the reviewers’ recognition of the paper’s strengths. Reviewer SJGk highlights the consistent improvements across benchmarks and model scales, the component-isolating ablations, and the value of questioning coordinate-token prediction as the default formulation. Reviewer aJTK recognizes the new decoder design with Crosshair Attention and layer-wise supervision, the extensive evaluation across model scales and benchmarks, and the manually curated Atypical-GUI-Bench. Reviewer MXDU considers the detector-style formulation and its internal iterative refinement as novel and intuitive, and finds the empirical gains over GUI-Actor and the supporting ablations convincing. We greatly appreciate these positive assessments. Our main responses for reviewers' concerns and new results are summarized below.

  1. Practical value in GUI agents. We clarify that planning and grounding are complementary. New OSWorld experiments with Hy4, Opus 5.5, and Sonnet 5.5 show that IntentLoc consistently improves strong planners while often reducing planner-side tokens and task latency.
  2. Scope of the semantic-ranking diagnosis. We refine our claim from a universal bottleneck of grounding model to a major limitation of conventional detectors. A controlled comparison between GroundingDINO-Swin-T and MM-GroundingDINO-Swin-L shows that increasing detector capacity only modestly improves ranking, whereas OCR-based semantic reranking yields large gains for both models.
  3. Direct evidence that IntentLoc addresses semantic ranking. New ScreenSpot-Pro Top-k analysis shows rapid saturation after Top-1, while the same OCR reranker improves IntentLoc by only 1.01 points. This indicates that IntentLoc already ranks the correct candidate first in nearly all cases where it is recalled.
  4. Direct evidence of intent and task reasoning. We clarify that the training corpus covers target-span, single-step intent-level, and task-level instructions. New Mind2Web and AndroidControl experiments show substantial gains under task-level and high-level prompts, where the target cannot be recovered through simple lexical matching.
  5. Scalability to newer VLMs. We add Qwen3-VL-8B and Qwen3.5-4B variants and compare them with recent same-backbone grounders. The compact Qwen3.5-4B variant ranks first or second on all four reported grounding benchmarks.
  6. Source of the coarse-to-fine gains. We distinguish IntentLoc’s internal layer-wise refinement from external iterative crop/zoom. New GUI-Actor zoom/crop baselines, and the decoder-layer ablation jointly show that both the detector-style objective and internal coarse-to-fine refinement contribute to the improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.