- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Seg2Click predicts a clickable mask instead of directly generating a coordinate or assigning one score to each visual patch. It uses a pixel-shuffle decoder for finer spatial predictions and converts the largest predicted component into a click.
Critical strengths
- Predicting below the visual-token resolution is a sensible way to handle small and crowded controls.
- The method works in one pass and does not require a verifier or repeated zooming.
- The comparison with coordinate generation uses a reasonably controlled setup.
Critical weaknesses
- The architecture is fairly close to existing multimodal segmentation heads (MLLMSeg), so the methodological novelty is moderate.
- The most important decoder ablation is missing. Pixel shuffle should be compared with simpler upsampling or transposed convolution under the same features and loss.
- The mask-to-click rule is fixed without comparison to alternatives such as selecting the highest-confidence point.
- Filled box supervision may be a poor approximation for irregular or partly occluded controls. The paper also does not test whether better grounding improves an interactive agent.
Questions for authors
see weakness
W1. The architecture is fairly close to existing multimodal segmentation heads (MLLMSeg), so the methodological novelty is moderate.
Our core innovation is redefining the GUI grounding task, rather than introducing a new network architecture. Seg2Click formulates grounding as box-supervised executable-region prediction: learning where an instruction can be executed, then computing a single click from the predicted region using a deterministic rule. Unlike MLLMSeg, which provides related segmentation machinery, Seg2Click defines the GUI action target as an executable region. This changes the prediction target and supervision from a coordinate sequence or visual-token scores to the spatial extent of the target control.
W2. The most important decoder ablation is missing. Pixel shuffle should be compared with simpler upsampling or transposed convolution under the same features and loss.
We thank the reviewer for suggesting this decoder ablation. We compared pixel shuffle with bilinear interpolation, resize-convolution, and transposed convolution on Qwen3.5-0.8B. Across the four variants, we kept the visual features, action query, box-supervised BCE+Dice loss, training data, training schedule, and centroid-based click extraction fixed, changing only the final upsampler. Each variant received 39,226 optimizer updates under the same two-stage training schedule.
All four variants achieve comparable performance, including the simpler bilinear upsampler. These results show that pixel shuffle is not essential to the observed performance. The contribution is the task formulation: learning a spatial target from box supervision and converting the predicted region into one click. This supplies supervision over the target's spatial extent instead of a single coordinate, and remains effective across the tested decoder choices.
W3. The mask-to-click rule is fixed without comparison to alternatives such as selecting the highest-confidence point.
We thank the reviewer for pointing out the need to compare the fixed centroid rule with alternatives. In particular, we tested the suggested highest-confidence point by selecting the pixel with the maximum predicted score, and compared it with four other click rules on common-protocol re-evaluations of the same saved 0.8B prediction fields, without retraining. Rules that use a connected component apply the same threshold of 0.5.
Selecting the point farthest from the component boundary raises ScreenSpot-v2 accuracy from 89.86% to 90.09% and ScreenSpot-Pro accuracy from 44.53% to 45.29%, while ScreenSpot changes by only −0.08 percentage points. These results show that better mask-to-click rules can further improve grounding accuracy without retraining the model.
W4. Filled box supervision may be a poor approximation for irregular or partly occluded controls. The paper also does not test whether better grounding improves an interactive agent.
Widely used GUI grounding datasets provide target boxes rather than pixel-level executable-region annotations, limiting the precision of available supervision. We evaluated whether Seg2Click remains effective under this imperfect supervision in closed-loop BrowserGym MiniWoB++, using the same screenshot-only Qwen3.5-4B planner, action interface, 15-step budget, and ten paired seeds per task; only the Qwen3.5-0.8B grounder varied.
Despite the annotation limitation, Seg2Click achieves 84.0% success on the full suite, 4.0 percentage points above GUI-Actor. On hard-target tasks, it achieves the highest success rate (67.5%), exceeding coordinate generation by 5.0 points and GUI-Actor by 2.5 points. Its advantage under this imperfect supervision demonstrates the effectiveness of the proposed formulation in executed GUI tasks, particularly for challenging targets.
Ethics flag
The review includes an ethics flag but does not specify a concern; we would appreciate clarification so we can address it directly.