acceptodds
Under review as a conference paper at ICLR 2027

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

Abstract

Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot. Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations. Region-based patch scoring alleviates this ambiguity, but its spatial decisions remain tied to visual-token footprints that may span the target, neighboring controls, and background. To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction. Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule. This formulation captures the spatial extent of valid click locations and enables finer localization beyond the visual-token grid, making it well suited to small, densely arranged controls without requiring a verifier or zoom-in pass. Experiments across five backbone configurations from two model families demonstrate the effectiveness of Seg2Click. Seg2Click-9B achieves 92.06%, 94.10%, and 55.22% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Visual Grounding, Vision-Language Models, Box-Supervised Action Masks, Sub-Token Spatial Prediction, GUI Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YosvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Seg2Click predicts a clickable mask instead of directly generating a coordinate or assigning one score to each visual patch. It uses a pixel-shuffle decoder for finer spatial predictions and converts the largest predicted component into a click.

Critical strengths
  1. Predicting below the visual-token resolution is a sensible way to handle small and crowded controls.
  2. The method works in one pass and does not require a verifier or repeated zooming.
  3. The comparison with coordinate generation uses a reasonably controlled setup.
Critical weaknesses
  1. The architecture is fairly close to existing multimodal segmentation heads (MLLMSeg), so the methodological novelty is moderate.
  2. The most important decoder ablation is missing. Pixel shuffle should be compared with simpler upsampling or transposed convolution under the same features and loss.
  3. The mask-to-click rule is fixed without comparison to alternatives such as selecting the highest-confidence point.
  4. Filled box supervision may be a poor approximation for irregular or partly occluded controls. The paper also does not test whether better grounding improves an interactive agent.
Questions for authors

see weakness

1 comment

W1. The architecture is fairly close to existing multimodal segmentation heads (MLLMSeg), so the methodological novelty is moderate.

Our core innovation is redefining the GUI grounding task, rather than introducing a new network architecture. Seg2Click formulates grounding as box-supervised executable-region prediction: learning where an instruction can be executed, then computing a single click from the predicted region using a deterministic rule. Unlike MLLMSeg, which provides related segmentation machinery, Seg2Click defines the GUI action target as an executable region. This changes the prediction target and supervision from a coordinate sequence or visual-token scores to the spatial extent of the target control.

W2. The most important decoder ablation is missing. Pixel shuffle should be compared with simpler upsampling or transposed convolution under the same features and loss.

We thank the reviewer for suggesting this decoder ablation. We compared pixel shuffle with bilinear interpolation, resize-convolution, and transposed convolution on Qwen3.5-0.8B. Across the four variants, we kept the visual features, action query, box-supervised BCE+Dice loss, training data, training schedule, and centroid-based click extraction fixed, changing only the final upsampler. Each variant received 39,226 optimizer updates under the same two-stage training schedule.

UpsamplerScreenSpot (%)ScreenSpot-v2 (%)ScreenSpot-Pro (%)
Bilinear interpolation87.1989.3143.83
Resize + convolution86.7989.3942.44
Transposed convolution87.5090.0944.47
Pixel shuffle87.1189.6242.76

All four variants achieve comparable performance, including the simpler bilinear upsampler. These results show that pixel shuffle is not essential to the observed performance. The contribution is the task formulation: learning a spatial target from box supervision and converting the predicted region into one click. This supplies supervision over the target's spatial extent instead of a single coordinate, and remains effective across the tested decoder choices.

W3. The mask-to-click rule is fixed without comparison to alternatives such as selecting the highest-confidence point.

We thank the reviewer for pointing out the need to compare the fixed centroid rule with alternatives. In particular, we tested the suggested highest-confidence point by selecting the pixel with the maximum predicted score, and compared it with four other click rules on common-protocol re-evaluations of the same saved 0.8B prediction fields, without retraining. Rules that use a connected component apply the same threshold of 0.5.

Click ruleScreenSpot (%)ScreenSpot-v2 (%)ScreenSpot-Pro (%)
Largest-component centroid87.5889.8644.53
Global maximum-confidence pixel86.7989.4744.47
Maximum-confidence pixel within component86.6489.1544.59
Probability-weighted component centroid87.5890.0244.85
Maximum-distance-to-boundary point within component87.5090.0945.29

Selecting the point farthest from the component boundary raises ScreenSpot-v2 accuracy from 89.86% to 90.09% and ScreenSpot-Pro accuracy from 44.53% to 45.29%, while ScreenSpot changes by only −0.08 percentage points. These results show that better mask-to-click rules can further improve grounding accuracy without retraining the model.

W4. Filled box supervision may be a poor approximation for irregular or partly occluded controls. The paper also does not test whether better grounding improves an interactive agent.

Widely used GUI grounding datasets provide target boxes rather than pixel-level executable-region annotations, limiting the precision of available supervision. We evaluated whether Seg2Click remains effective under this imperfect supervision in closed-loop BrowserGym MiniWoB++, using the same screenshot-only Qwen3.5-4B planner, action interface, 15-step budget, and ten paired seeds per task; only the Qwen3.5-0.8B grounder varied.

Grounder20-task suite (200 episodes)Hard-target suite (120 episodes)
Coordinate generation169/200 (84.5%)75/120 (62.5%)
GUI-Actor160/200 (80.0%)78/120 (65.0%)
Seg2Click168/200 (84.0%)81/120 (67.5%)

Despite the annotation limitation, Seg2Click achieves 84.0% success on the full suite, 4.0 percentage points above GUI-Actor. On hard-target tasks, it achieves the highest success rate (67.5%), exceeding coordinate generation by 5.0 points and GUI-Actor by 2.5 points. Its advantage under this imperfect supervision demonstrates the effectiveness of the proposed formulation in executed GUI tasks, particularly for challenging targets.

Ethics flag

The review includes an ethics flag but does not specify a concern; we would appreciate clarification so we can address it directly.

Reviewer_MD5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present a method combines visual encoder features and language-model representations, conditions a pixel-shuffle decoder on an instruction-aware action query, and predicts dense action masks supervised by target bounding boxes. A deterministic largest-connected-component centroid rule converts the predicted mask into a click coordinate.

Critical strengths
  • A well-motivated output representation for fine-grained GUI grounding.
  • Strong benchmark results with useful controlled comparisons.
Critical weaknesses

Although the paper provides useful controlled comparisons and decoder-width ablations, the direct-coordinate, patch-scoring, and mask-based architectures differ in their trainable heads and optimization objectives.

The predicted masks are supervised by rectangular bounding boxes, which are approximations of valid click regions rather than actual executable areas. Evaluation relies exclusively on static box-hit accuracy, without measuring real interface execution success.

Exps show that the largest-component centroid can fall outside the intended control even when the predicted mask overlaps it. -> the evidence does not show whether the formulation improves real-world GUI interaction robustness beyond benchmark localization.

Questions for authors

See weakness

1 comment

W1. The three formulations differ in heads and objectives, so the comparison is not causal

Table 6 is a comparison of complete task formulations, not an attempt to isolate a representation from its head and loss. Each formulation requires its corresponding supervision: coordinate generation predicts a token sequence, patch scoring selects visual tokens, and Seg2Click predicts a box-supervised spatial region. Applying the same head and loss to all three would change the formulations being compared. We therefore match the backbone and training data and compare each method in its intended form. Our claim is that redefining GUI grounding as box-supervised region prediction produces a stronger grounding method, not that the mask representation can be evaluated independently of its objective.

W2. Box supervision approximates the executable region, and evaluation is static box-hit only

We evaluated closed-loop task completion in BrowserGym's MiniWoB++ environment. All methods used the same screenshot-only Qwen3.5-4B planner, a 15-step budget, and 10 paired seeds per task; only the Qwen3.5-0.8B grounder changed. We tested both the 20-task suite and a 12-task hard-target subset covering small, dense, or irregular targets.

MethodMiniWoB++ 20-task successMiniWoB++ hard-target successScreenSpot latencyScreenSpot-v2 latencyScreenSpot-Pro latency
Coordinate generation169/200 (84.5%)75/120 (62.5%)627.7 ms605.7 ms679.3 ms
GUI-Actor160/200 (80.0%)78/120 (65.0%)230.1 ms209.4 ms370.2 ms
Seg2Click168/200 (84.0%)81/120 (67.5%)219.3 ms192.8 ms345.4 ms

Seg2Click achieved the highest success rate on the hard-target MiniWoB++ subset, outperforming coordinate generation and GUI-Actor, and completed more tasks than GUI-Actor on the full suite. It also had the lowest inference latency among the three methods on all benchmarks, including a modest advantage over GUI-Actor. These results show that Seg2Click combines effective grounding for challenging targets with fast single-pass inference; static box-hit evaluation provides a reproducible benchmark, while MiniWoB++ tests task completion after actions are executed.

W3. The centroid can fall outside the intended control even when the mask overlaps it

In Figure 4, the predicted mask extends onto adjacent controls or labels, whose pixels shift the largest-component centroid outside the target despite partial overlap. This reflects inaccurate region prediction in individual examples, rather than evidence against the region-prediction formulation. The results above show that Seg2Click improves closed-loop task completion on challenging targets while achieving faster inference than both coordinate generation and GUI-Actor.

Reviewer_RcHkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Seg2Click, a vision-based GUI grounding framework that reformulates coordinate prediction as pixel-level action mask prediction. The authors argue that coordinate generation methods suffer from ambiguous point supervision, while patch-based grounding approaches are limited by the spatial granularity of visual tokens. Seg2Click introduces a query-conditioned spatial decoder that fuses vision encoder features with language-model representations. The decoder employs convolutional layers and pixel shuffle to generate a dense action mask, supervised by binary masks derived from existing bounding-box annotations. During inference, the final click coordinate is determined by computing the centroid of the largest connected component in the predicted mask. Seg2Click is built upon Qwen2.5-VL and Qwen3.5 series, and show improved accuracy on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks.

Critical strengths
  1. Seg2Click provides a simple and practical approach to GUI grounding by predicting action masks instead of directly generating click coordinates. 2.The experiments cover three benchmarks and multiple backbones. Seg2Click also outperforms coordinate generation and GUI-Actor on ScreenSpot-Pro under the same Qwen3.5-0.8B backbone.
Critical weaknesses
  1. Regarding the novelty of this work, given the substantial architectural similarities to MLLMSeg and PixelLM, what are the specific contributions of Seg2Click beyond adapting an existing MLLM segmentation framework to GUI grounding? 2.I can see Seg2Click’s improvements over existing methods on several GUI grounding benchmarks. However, it is not evaluated GUI agent tasks. It remains unclear whether these improvements in grounding accuracy translate into better performance in real-world GUI agent scenarios. Previous works, such as SeeClick, have also evaluated their methods on downstream GUI agent tasks. 3.The contribution of sub-token prediction is not fully isolated. A comparison with a simpler upsampling-based mask decoder under the same settings would help clarify whether the proposed decoder is responsible for the improvements.
Questions for authors
  1. Scaling Seg2Click from 2B to 4B improves ScreenSpot-Pro overall but loses 6 and 9 correct clicks on the smallest and the largest targets (Appendix B.1). Do you have an explanation about this anomaly?
  2. The paper reveals the inference cost of Seg2Click on one H200. Could the authors provide inference costs of other coordinate generation methods and patch-based grounding methods for a fair comparison?
  3. Can the authors compare the proposed sub-token prediction with a simpler upsampling-based approach under the same training settings, to better demonstrate the benefit of the proposed decoder?
1 comment

CW1: Contribution beyond MLLMSeg / PixelLM

We do not claim architectural novelty. To our knowledge, Seg2Click is the first to formulate GUI grounding as box-supervised executable-region prediction, with a fixed rule that derives a single click from the predicted mask. Unlike direct coordinate regression or visual-token scoring, this formulation learns a spatial target from existing GUI box annotations. MLLMSeg and PixelLM provide related segmentation machinery, but do not address this GUI grounding formulation.

CW2: Not evaluated on GUI agent tasks

We evaluated downstream task success in a closed-loop BrowserGym/MiniWoB++ agent. All methods use the same screenshot-only Qwen3.5-4B planner, action interface, and 15-step budget. We replace only the Qwen3.5-0.8B grounder, with ten paired initial seeds per task type. The evaluation covers 20 task types (200 episodes per method), plus a separate set of 12 task types selected for small, dense, or irregular targets (120 episodes per method), including click-tab-2-hard, click-checkboxes-soft, and click-pie. The metric is episode-level task success after executing the agent's actions.

Grounder20-task suite (200 episodes)Hard-target suite (120 episodes)
Coordinate generation169/200 (84.5%)75/120 (62.5%)
GUI-Actor160/200 (80.0%)78/120 (65.0%)
Seg2Click168/200 (84.0%)81/120 (67.5%)

On the 20-task suite, Seg2Click improves over GUI-Actor by 4.0 percentage points. On the hard-target suite, Seg2Click improves over coordinate generation by 5.0 percentage points, completing six additional paired episodes. Relative to GUI-Actor, it gains 11 successful episodes and loses eight, for a net improvement of 2.5 percentage points.

CW3 / Q3: Is the sub-token decoder responsible for the gains?

To test whether pixel shuffle itself is necessary for the observed performance, we compared it with bilinear interpolation, resize-convolution, and transposed convolution on Qwen3.5-0.8B. We kept the visual features, training loss, training settings, and rule for selecting the final click the same across all four models.

UpsamplerScreenSpot (%)SS-v2 (%)SS-Pro (%)
Bilinear interpolation87.1989.3143.83
Resize + convolution86.7989.3942.44
Transposed convolution87.5090.0944.47
Pixel shuffle87.1189.6242.76

Transposed convolution achieves the highest accuracy on all three benchmarks. Bilinear interpolation also performs competitively, scoring 43.83% on ScreenSpot-Pro compared with 42.76% for pixel shuffle. These results show that performance does not depend on pixel shuffle. The main contribution is instead how we formulate GUI grounding: the model predicts a spatial mask from box supervision, then a fixed rule uses that mask to select one click point. The decoder is one way to produce the mask, not the main contribution.

Q1: Why does 2B→4B gain overall but lose 6 and 9 clicks on the smallest and largest targets?

The 4B model improves overall by six correct clicks, with a gain of 21 on intermediate-sized targets and declines of six and nine on the smallest and largest targets, respectively. We attribute this non-uniform pattern to differences between the pretrained backbones and their response to fine-tuning. Although larger models tend to have stronger overall capabilities, scaling does not improve every capability equally. GUI grounding requires target recognition, instruction understanding, and spatial localization, and gains in these abilities need not be uniform across target sizes. Thus, the overall gain at 4B does not imply gains in every size group.

Q2: Fair inference costs

We measured all three Qwen3.5-0.8B methods on the same H200 using the same 200 examples, batch size, and pixel bounds. Timing includes preprocessing, inference, parsing, and click extraction; model loading and image decoding are excluded.

BenchmarkCoordinate gen.GUI-ActorSeg2Click
ScreenSpot627.7 ms230.1 ms219.3 ms
ScreenSpot-v2605.7 ms209.4 ms192.8 ms
ScreenSpot-Pro679.3 ms370.2 ms345.4 ms

Coordinate generation is slowest because it autoregressively generates an average of 23 coordinate tokens. GUI-Actor and Seg2Click use the same Qwen3.5-0.8B backbone and both produce their spatial outputs in one pass, so their shared vision-language computation keeps their runtimes close. Their heads differ: GUI-Actor scores visual patches with pointer attention, while Seg2Click predicts a spatial mask with a convolutional decoder and selects its centroid. Seg2Click is slightly faster on all three benchmarks.

Comments on the paper

1 comment
Area_Chair_DPGaOpenReview ↗

Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.