- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies errors in GUI grounding. It argues that standard point accuracy hides an important distinction between clicking background regions and clicking incorrect GUI elements. They observe substantial agreement across models on which incorrect elements are selected, suggesting that these elements constitute systematic distractors rather than random localization errors. The paper proposes CONCLICK, which explicitly trains a grounding model to distinguish the target from hard distractor elements. The authors construct 4,292 target-distractor pairs from Click100K using incorrect predictions shared by GTA1-7B and GUI-Libra-8B, and combine coordinate supervision with a listwise ranking objective and a pairwise margin objective over attention-derived region scores. On Qwen3-VL-8B, CONCLICK improves accuracy from 54.71% to 66.03% on ScreenSpot-Pro, 63.33% to 70.20% on OSWorld-G, and 83.72% to 86.28% on MMBench-GUI L2. RECLICK further raises the corresponding accuracies. Ablations indicate that both the hard-negative data and contrastive losses contribute, and experiments on InternVL3.5-8B and Qwen3-VL-30B-A3B-Instruct suggest that the training idea is not specific to the main 8B backbone.
Critical strengths
-
The diagnostic analysis leverages a concrete and actionable failure mode rather than merely proposing another grounding objective. The analysis covers 10 models and five datasets (161,640 predictions), and shows both that incorrect clicks disproportionately land on GUI elements and that different models frequently select the same distractors. I view this as the strongest contribution of the paper, though similar conclusions have been shown in similar papers on single models.
-
The proposed method is well aligned with the diagnosis and is supported by reasonably thorough experiments. The gain is meaningful, and the additional-backbone experiments point in the same direction, which reduces the concern that the result is specific to Qwen3-VL-8B.
Critical weaknesses
- The central error taxonomy and hard-negative pipeline depend on weak external components, and the validation of this dependency is somewhat limited. Element errors are defined using OmniParser-v2 detections, so missed or spurious elements can change the measured error distribution. The training pairs are further selected using agreement between two grounding models and then filtered by Gemini 3.5 Flash. How do these choices affect the performance of CONCLICK?
Questions for authors
-
How much training variance is there for CONCLICK? Could the authors provide results across multiple random seeds for at least the Qwen3-VL-8B baseline?
-
What is the practical inference cost of RECLICK relative to direct grounding and the inference baselines? Since part of its gain comes from additional inference computation, wall-clock latency or token cost for direct inference would make the comparison easier to interpret.
2 comments
Follow-up on External Components and Inference Cost
Dear Reviewer xyUX,
Just a friendly reminder as the discussion deadline approaches. We hope our clarifications have addressed your concerns.
We have supplemented the appendix with parser clarifications, Gemini filtering statistics, training variance results, and practical inference costs.
We look forward to hearing from you and would gladly discuss anything that remains unclear. Thank you for your time and thoughtful feedback!
Response to Reviewer xyUX
We thank the reviewer for the positive assessment of our diagnostic analysis, the connection to our method, and the gains across model backbones. We address the questions about external components, training variance, and inference cost below, and would be happy to discuss these points further or answer any additional questions.
Thank you for this great suggestion. We address this concern through manual validation, controlled data comparisons, and the filtering statistics below.
(1) OmniParser: manual validation. In Appendix A.1, 87 of 100 randomly sampled screenshots met the detection-quality criterion, and 92 of 100 sampled non-target clicks were manually confirmed. These checks support the reliability of the sampled detections and labels. Missed detections that cause genuine element errors to be classified as background errors would underestimate, rather than inflate, the prevalence of element errors. Importantly, our primary metric, point accuracy, is evaluated against the original benchmark target annotations, independently of parser-based error categorization. We observe consistent accuracy improvements across all three benchmarks and multiple model backbones (Tables 1, 2, and A2).
(2) Two-model agreement: comparison with random sampling. With the training-set size fixed at 8,584 examples and the same coordinate-only objective, ConClick-Data improves mean accuracy by 1.26 percentage points over randomly sampled Click100K data across three training runs. This supports the effectiveness of the constructed data. Table A2 further reports data gains on two additional backbones across all three benchmarks.
(3) Gemini filtering: a limited annotation-quality check. Gemini excluded only 13 of 4,305 candidate pairs (0.30%), leaving 4,292 pairs for training. Manual inspection found that the excluded cases mainly involved incorrect original Click100K annotations. Thus, this check changed only a small fraction of the mined dataset.
We will include these clarifications and filtering statistics in the revised paper.
Great question. We now report three independent training runs for each trainable Qwen3-VL-8B setting in Table 2. Within each setting, all runs start from the same pretrained checkpoint, with the training data, hyperparameters, and checkpoint-selection rule held fixed. The table reports macro-average accuracy (%) across the three equally weighted benchmarks, with mean ± sample standard deviation across training runs.
Across three runs, the full ConClick objective achieves 74.04 ± 0.24% macro-average accuracy, retaining a 2.17-point mean gain over coordinate-only training on the same data and a 3.43-point mean gain over the coordinate-only baseline trained on an equally sized random subset. We will include these multi-seed results in the revised paper.
Great suggestion. We compare inference cost with direct grounding using the same ConClick-8B checkpoint. We also increase MVP's view budget so its FLOPs meet or exceed ReClick's on each benchmark.
Values are unweighted averages across the three benchmarks. FLOPs are profiled on the full submitted evaluation subsets. Latency is measured on NVIDIA H200 GPUs (BF16, one query at a time), using the same fixed 100 examples per benchmark and two timed repetitions after warm-up. Timing includes preprocessing, model inference, cropping and candidate selection, excluding model loading and source-image I/O.
Compared with direct grounding, ReClick improves average accuracy by 2.96 percentage points, while increasing mean latency from 0.89 to 4.15 seconds per query. ReClick outperforms higher-budget MVP by 1.68 percentage points on average, with gains on all three benchmarks, while using 6.1% fewer FLOPs and lower measured latency (4.15 vs. 4.84 s/query). Its candidate scoring reuses cached prefixes without re-encoding images.
We will include this cost–accuracy comparison in the revised paper.