acceptodds
Under review as a conference paper at ICLR 2027

ConClick: Element-wise Contrastive Learning Improves GUI Grounding

16 views

Abstract

GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents. However, existing grounding models remain error-prone, often causing downstream task failures. We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors. Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors. We construct ConClick-Data, a dataset containing 4,292 challenging target–distractor pairs, and train the model using complementary listwise ranking and pairwise margin objectives. ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones. We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks. Together, our analysis, dataset, and methods provide both empirical insights and practical tools for building more reliable GUI agents. Data and code are available at https://anonymous.4open.science/r/ConClick-review-B321

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Grounding, GUI Agents, Contrastive Learning, Error Analysis, Test-Time Refinement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xyUXOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies errors in GUI grounding. It argues that standard point accuracy hides an important distinction between clicking background regions and clicking incorrect GUI elements. They observe substantial agreement across models on which incorrect elements are selected, suggesting that these elements constitute systematic distractors rather than random localization errors. The paper proposes CONCLICK, which explicitly trains a grounding model to distinguish the target from hard distractor elements. The authors construct 4,292 target-distractor pairs from Click100K using incorrect predictions shared by GTA1-7B and GUI-Libra-8B, and combine coordinate supervision with a listwise ranking objective and a pairwise margin objective over attention-derived region scores. On Qwen3-VL-8B, CONCLICK improves accuracy from 54.71% to 66.03% on ScreenSpot-Pro, 63.33% to 70.20% on OSWorld-G, and 83.72% to 86.28% on MMBench-GUI L2. RECLICK further raises the corresponding accuracies. Ablations indicate that both the hard-negative data and contrastive losses contribute, and experiments on InternVL3.5-8B and Qwen3-VL-30B-A3B-Instruct suggest that the training idea is not specific to the main 8B backbone.

Critical strengths
  1. The diagnostic analysis leverages a concrete and actionable failure mode rather than merely proposing another grounding objective. The analysis covers 10 models and five datasets (161,640 predictions), and shows both that incorrect clicks disproportionately land on GUI elements and that different models frequently select the same distractors. I view this as the strongest contribution of the paper, though similar conclusions have been shown in similar papers on single models.

  2. The proposed method is well aligned with the diagnosis and is supported by reasonably thorough experiments. The gain is meaningful, and the additional-backbone experiments point in the same direction, which reduces the concern that the result is specific to Qwen3-VL-8B.

Critical weaknesses
  1. The central error taxonomy and hard-negative pipeline depend on weak external components, and the validation of this dependency is somewhat limited. Element errors are defined using OmniParser-v2 detections, so missed or spurious elements can change the measured error distribution. The training pairs are further selected using agreement between two grounding models and then filtered by Gemini 3.5 Flash. How do these choices affect the performance of CONCLICK?
Questions for authors
  1. How much training variance is there for CONCLICK? Could the authors provide results across multiple random seeds for at least the Qwen3-VL-8B baseline?

  2. What is the practical inference cost of RECLICK relative to direct grounding and the inference baselines? Since part of its gain comes from additional inference computation, wall-clock latency or token cost for direct inference would make the comparison easier to interpret.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xyUX

We thank the reviewer for the positive assessment of our diagnostic analysis, the connection to our method, and the gains across model backbones. We address the questions about external components, training variance, and inference cost below, and would be happy to discuss these points further or answer any additional questions.


W1: Dependence on external components in error analysis and hard-negative mining.

Thank you for this great suggestion. We address this concern through manual validation, controlled data comparisons, and the filtering statistics below.

(1) OmniParser: manual validation. In Appendix A.1, 87 of 100 randomly sampled screenshots met the detection-quality criterion, and 92 of 100 sampled non-target clicks were manually confirmed. These checks support the reliability of the sampled detections and labels. Missed detections that cause genuine element errors to be classified as background errors would underestimate, rather than inflate, the prevalence of element errors. Importantly, our primary metric, point accuracy, is evaluated against the original benchmark target annotations, independently of parser-based error categorization. We observe consistent accuracy improvements across all three benchmarks and multiple model backbones (Tables 1, 2, and A2).

(2) Two-model agreement: comparison with random sampling. With the training-set size fixed at 8,584 examples and the same coordinate-only objective, ConClick-Data improves mean accuracy by 1.26 percentage points over randomly sampled Click100K data across three training runs. This supports the effectiveness of the constructed data. Table A2 further reports data gains on two additional backbones across all three benchmarks.

(3) Gemini filtering: a limited annotation-quality check. Gemini excluded only 13 of 4,305 candidate pairs (0.30%), leaving 4,292 pairs for training. Manual inspection found that the excluded cases mainly involved incorrect original Click100K annotations. Thus, this check changed only a small fraction of the mined dataset.

We will include these clarifications and filtering statistics in the revised paper.


Q1: Training variance across random seeds.

Great question. We now report three independent training runs for each trainable Qwen3-VL-8B setting in Table 2. Within each setting, all runs start from the same pretrained checkpoint, with the training data, hyperparameters, and checkpoint-selection rule held fixed. The table reports macro-average accuracy (%) across the three equally weighted benchmarks, with mean ± sample standard deviation across training runs.

Training settingRun 1Run 2Run 3Mean ± SD
Random data + coordinate loss70.6770.7070.4470.61 ± 0.15
ConClick-Data + coordinate loss71.9972.0271.5971.87 ± 0.24
ConClick-Data + coordinate + listwise loss73.4273.2873.3473.35 ± 0.07
ConClick-Data + coordinate + pairwise loss73.1873.0773.3673.20 ± 0.14
ConClick-Data + full objective74.1773.7674.1874.04 ± 0.24

Across three runs, the full ConClick objective achieves 74.04 ± 0.24% macro-average accuracy, retaining a 2.17-point mean gain over coordinate-only training on the same data and a 3.43-point mean gain over the coordinate-only baseline trained on an equally sized random subset. We will include these multi-seed results in the revised paper.


Q2: Practical inference cost.

Great suggestion. We compare inference cost with direct grounding using the same ConClick-8B checkpoint. We also increase MVP's view budget so its FLOPs meet or exceed ReClick's on each benchmark.

MethodFLOPs (T/query)Mean latency (s/query)Avg. accuracy (%)
Direct101.420.8974.17
MVP (paper setting)350.063.3475.71
MVP (increased budget)477.964.8475.45
ReClick448.654.1577.13

Values are unweighted averages across the three benchmarks. FLOPs are profiled on the full submitted evaluation subsets. Latency is measured on NVIDIA H200 GPUs (BF16, one query at a time), using the same fixed 100 examples per benchmark and two timed repetitions after warm-up. Timing includes preprocessing, model inference, cropping and candidate selection, excluding model loading and source-image I/O.

Compared with direct grounding, ReClick improves average accuracy by 2.96 percentage points, while increasing mean latency from 0.89 to 4.15 seconds per query. ReClick outperforms higher-budget MVP by 1.68 percentage points on average, with gains on all three benchmarks, while using 6.1% fewer FLOPs and lower measured latency (4.15 vs. 4.84 s/query). Its candidate scoring reuses cached prefixes without re-encoding images.

We will include this cost–accuracy comparison in the revised paper.

AuthorsOpenReview ↗

Follow-up on External Components and Inference Cost

Dear Reviewer xyUX,

Just a friendly reminder as the discussion deadline approaches. We hope our clarifications have addressed your concerns.

We have supplemented the appendix with parser clarifications, Gemini filtering statistics, training variance results, and practical inference costs.

We look forward to hearing from you and would gladly discuss anything that remains unclear. Thank you for your time and thoughtful feedback!

Reviewer_mCoLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies GUI grounding errors across ten models and five datasets. It finds that many wrong clicks land on another GUI element. ConClick trains the model to distinguish the target from these distractors using listwise ranking loss and pairwise margin loss, together with coordinate loss. ReClick makes further predictions on enlarged crops. It then chooses a click using cross-view support and spatial agreement. The paper tests both methods on three benchmarks.

Critical strengths
  1. The error analysis is broad and checked in several ways. It covers ten models and five datasets. The authors compare wrong clicks with a random-click reference and manually check some detected elements and error labels (Section 2; Appendix A). These checks support the finding that choosing the wrong element is a common error.
  2. The loss improves results with the training data held fixed. Average accuracy rises from 71.99% to 74.17% in Table 2. Table A2 also reports gains on two other models. This is useful evidence that the loss helps, beyond the benefit of changing the training data.
Critical weaknesses
  1. Some gains are small, and their reliability is unclear. The full loss improves on listwise ranking loss alone by 0.75 percentage points in Table 2. Repeated training runs and confidence intervals for these differences are not reported. These results do not yet show that every part of the design gives a reliable gain.
  2. The ReClick comparison does not use clearly matched computing budgets. ReClick uses up to four visual grounding passes and extra scoring of candidate coordinates, called cached-prefix candidate continuations in the paper. The MVP setting used with ConClick uses up to two crop predictions (Appendix C). Table A3 shows higher accuracy under the tested settings, but it does not show which method is better when both use the same amount of computation.
Questions for authors
  1. Clarification: did all training versions in Table 2 use the same training stages, model-selection rules, and amount of hyperparameter search?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mCoL

We thank the reviewer for the kind words about our error analysis and the benefits of contrastive supervision, as well as the thoughtful questions. We address each point below and would be happy to provide further clarification or answer any additional questions.


W1: Reliability of the gains and training variance.

Thank you for raising this point. We now report three independent training runs for each trainable setting in Table 2. Within each setting, all runs start from the same pretrained checkpoint, with the training data, hyperparameters, and checkpoint-selection rule held fixed. The three contrastive variants use matched training seeds. Results below are macro-average accuracy (%) across the three equally weighted benchmarks, with mean ± sample standard deviation across runs.

Training settingRun 1Run 2Run 3Mean ± SD
Random data + coordinate loss70.6770.7070.4470.61 ± 0.15
Our data + coordinate loss71.9972.0271.5971.87 ± 0.24
Our data + coordinate + listwise loss73.4273.2873.3473.35 ± 0.07
Our data + coordinate + pairwise loss73.1873.0773.3673.20 ± 0.14
Our data + full objective74.1773.7674.1874.04 ± 0.24

For each of the three matched seeds, the full objective achieves the highest macro-average accuracy among the contrastive variants. Its mean gains over listwise-only and pairwise-only are 0.69 and 0.83 percentage points, respectively (paired 95% t confidence intervals: [0.23, 1.15] and [0.45, 1.22]). These results show consistent gains across the three tested seeds. The full objective also retains a 2.17-point mean gain over coordinate-only training on the same data.


W2: Comparison under comparable computational budgets.

Thank you for this great suggestion. We clarify the pass counts for the settings reported in Table A3 and add a comparison in which MVP uses at least as many FLOPs per example on average as ReClick on each benchmark, as follows.

Settings reported in the paper (Table A3). ReClick uses up to four grounding passes: one full-image prediction and three crop predictions. Its cached candidate scoring does not re-encode images. MVP uses up to three grounding passes (one full image + two additional views); the official SSP/MMB implementation also performs one separate full-image attention pass. Because image sizes and scoring costs differ, we additionally report newly profiled mean TFLOPs per example for each configuration, including visual encoding, decoding, attention extraction, and cached candidate scoring:

BenchmarkMVP TFLOPs (Table A3 setting)ReClick TFLOPs (Table A3 setting)
ScreenSpot-Pro574.42774.12
OSWorld-G124.11190.70
MMBench-GUI L2351.64381.12

Additional comparison with higher-budget MVP. To account for the different computational costs of the Table A3 settings, we increase only MVP’s official max_inferences to 4/4/3 for ScreenSpot-Pro, OSWorld-G, and MMBench-GUI L2, respectively. This allows up to 5/5/4 grounding passes, compared with ReClick’s four. MVP additionally performs one separate full-image attention pass on ScreenSpot-Pro and MMBench-GUI L2, as described above. Both methods use the same checkpoint and the paper’s evaluation subsets.

BenchmarkNew MVP TFLOPsReClick TFLOPsNew MVP Acc. (%)ReClick Acc. (%, Table A3)
ScreenSpot-Pro779.54774.1270.1572.36
OSWorld-G207.71190.7070.0071.76
MMBench-GUI L2446.62381.1286.2087.28

ReClick is 2.21, 1.76, 1.08 percentage points more accurate, respectively, while using fewer FLOPs than these higher-budget MVP settings.

These results show that ReClick’s accuracy advantage persists when MVP is given a comparable or higher FLOP budget. We will incorporate these comparisons and the computational cost details into the revised paper before the deadline.


Q1: Training stages, model selection, and hyperparameter-search budgets.

Yes. All trained variants in Table 2 use the same training stages. We save checkpoints at identical step intervals and select each variant’s best checkpoint on the same held-out Click100K validation set. The hyperparameter-search budget is also identical across variants, including the coordinate-only baselines. We will explicitly clarify these settings in the revised paper.

AuthorsOpenReview ↗

Update on Training Variance and Computational Budgets

Dear Reviewer mCoL,

As the discussion period is nearing its end, we wanted to gently follow up. We hope our response and additional experiments have addressed your concerns.

The revised appendix includes results across three training runs and comparisons in which MVP receives at least as many FLOPs as ReClick.

We would appreciate your thoughts and any further feedback. Thank you again for your helpful suggestions!

Reviewer_Poh1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why GUI grounding models click the wrong place and proposes ConClick to improve. An error analysis finds that most wrong clicks land on another GUI element rather than on background, and that different models often pick the same wrong element. Based on this, ConClick mines 4,292 target–distractor pairs from Click100K and fine-tunes Qwen with a coordinate loss plus an element-level ranking loss on attention-derived region scores. ReClick is a training-free inference step that re-decodes the click on attention-selected zoomed crops and picks the final point by cross-view likelihood and agreement. The paper shows improvements over multiple grounding benchmarks.

Critical strengths
  • A well-motivated and clearly written study.
  • The recipe is simple and cheap (about 8.6k examples, LoRA). And the training can still improve models that are already relatively strong enough at grounding further.
Critical weaknesses
  1. Need better ablations and analysis to isoldate the contribution from

    1. plain coordinate fine-tuning on random Click100K data

    2. switching to the mined hard-pair data

    3. ontrastive loss

    It seems that the 2nd and 3rd might have not contributed most to the gains.

  2. ReClick is largely separable from the contrastive training, and the comparison set is soft. Can consider more baselines or even naive methods to compare against given that there are already many attempts around this. For example, the simplest baseline, crop around the first click and re-predict, is missing, and other cited zoom/training-free test-time methods (e.g., RegionFocus, UI-Zoomer, GUI-RC) are not compared.

Questions for authors

Please see the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Poh1 — Part 1 (W1)

We thank the reviewer for the positive assessment of the motivation and clarity of our study and the simplicity and efficiency of our method, as well as the constructive questions and suggestions. We address each point below and would be happy to answer any further questions.


W1: Isolating the contributions of coordinate fine-tuning on random Click100K data, mined hard-pair data, and the contrastive loss.

Thank you for raising this point. Table 2 and Section 4.3 (p. 8) isolate these three contributions using equally sized training datasets (8,584 examples). We now supplement this ablation with three independent training runs per setting, keeping the data, hyperparameters, and checkpoint-selection rule fixed within each setting. Below we report macro-average accuracy (%) across the three benchmarks, with mean ± sample standard deviation for the trained models.

Training settingAvg. acc. (%)Δ Pretrained (pp)Δ Previous (pp)
Pretrained Qwen3-VL-8B (submitted result)67.25——
Random data + coordinate loss70.61 ± 0.15+3.36+3.36
ConClick-Data + coordinate loss71.87 ± 0.24+4.62+1.26
ConClick-Data + full objective74.04 ± 0.24+6.79+2.17

These results show consistent improvements at each of the three steps. First, coordinate fine-tuning on Click100K establishes a strong task-adapted baseline for evaluating our proposed components. Even the randomly sampled subset comes from a dataset curated specifically for GUI grounding. We carefully tuned this coordinate-only baseline using held-out validation performance, with the same hyperparameter-search budget and checkpoint-selection rule as the other variants.

Relative to this strong baseline, hard-pair mining and contrastive supervision provide an additional 3.43-point improvement. With the training-set size fixed at 8,584 examples, replacing random data with ConClick-Data improves accuracy by 1.26 points under the same coordinate-only objective. Holding ConClick-Data fixed, adding contrastive supervision yields a further 2.17-point gain. These controlled comparisons isolate the benefit of each proposed component beyond coordinate fine-tuning alone. We therefore respectfully disagree with this assessment: the two proposed components deliver consistent gains on top of a carefully tuned baseline, providing a further 3.43-point improvement beyond the initial 3.36-point gain from coordinate fine-tuning.

Table A2 (p. 18) also reports gains from both components across all three benchmarks on two additional backbones, further supporting the generalizability of our approach across model backbones. We will include the new multi-seed results in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Poh1 — Part 2 (W2)

W2: ReClick's relationship to contrastive training and the choice of inference baselines.

Great question. We address this point below.

W2(a): Connection between contrastive training and ReClick.

  • Complementary roles. ConClick and ReClick address target–distractor ambiguity at training and inference time, respectively. ConClick learns attention-derived scores that favor annotated targets over distractors (§3.1). ReClick revisits high-scoring regions at test time and compares candidate clicks through cross-view support and spatial agreement (§3.2).
  • Learned weights reused at inference. When paired with ConClick, ReClick selects crops using the layer/head weights learned through contrastive training, providing a concrete connection between the two methods. In Table A4 of the submitted paper (p. 19; also Figure 6(c)), we have evaluated this connection with the checkpoint held fixed: learned crop selection achieves 77.13% three-benchmark average accuracy, compared with 76.67% when the learned layer/head weights are replaced with a fixed attention readout (“w/o learned crop selection”), supporting the benefit of reusing the learned region preferences.
  • Transferability. ReClick can also be applied to frozen grounders by adapting the attention readout (App. C.2). In Figure 6(b) and Table A3 (p. 19), we have demonstrated its benefits on GTA1-7B and UI-TARS-1.5-7B without our contrastive training, showing its broader applicability.

ReClick thus complements ConClick at inference time and benefits from its learned region preferences, while remaining transferable to other grounders. We will incorporate this discussion into the revised paper.

W2(b): Comparison with additional inference methods.

Following your suggestion, we have evaluated crop around the first click and re-predict, RegionFocus, UI-Zoomer, and GUI-RC on all three backbones using the submitted paper's benchmark subsets.

Average accuracy (%) across the three benchmarks (unweighted mean). “New” denotes additional rebuttal experiments.

Inference methodResultsConClick-8BUI-TARS-1.5-7BGTA1-7B
Direct controlTable 174.1761.9567.97
MVPTable A375.7165.6372.37
ZoomClickTable A372.2562.6968.52
Crop around the first click and re-predictNew73.7863.7670.47
RegionFocusNew73.8762.9365.80
GUI-RC (64 samples)New72.0759.2468.51
UI-Zoomer, explicit bbox promptNew54.510.73†60.06
UI-Zoomer, original promptNew35.690.73†65.51
ReClickTable A377.1368.2873.49

ReClick achieves the highest average accuracy on all three backbones, outperforming the strongest compared baseline, MVP, by 1.42, 2.65, and 1.12 percentage points, respectively. It also exceeds the reviewer-suggested first-click crop baseline by 3.35, 4.52, and 3.02 points.

On ConClick-8B, we additionally compare against higher-budget MVP settings whose FLOPs meet or exceed ReClick’s on each benchmark. ReClick remains more accurate on all three benchmarks, with an average gain of 1.68 percentage points while using 6.1% fewer FLOPs.

We use the official released implementations with model-interface and coordinate adapters, retaining each method's inference procedure and sampling budget.

  • Crop around the first click and re-predict: We return the second prediction from ReClick's initial-click crop directly. Table A4's “w/o score-guided crops” still reranks candidates. ReClick improves over this baseline on all nine backbone–benchmark pairs.
  • RegionFocus: We use the official UI-TARS implementation, originally released for UI-TARS-7B-DPO, adapting its model interface to our three backbones while preserving the inference procedure.
  • GUI-RC: The official evaluation includes UI-TARS-1.5-7B. We use its released 64-sample voting implementation and code defaults, with model-specific prompt and coordinate adapters.
  • UI-Zoomer: The released method requires bounding-box predictions and provides no point-output configuration for UI-TARS-1.5-7B, GTA1-7B or ConClick. We therefore evaluate two adaptations: original prompt, using the official bbox request; and explicit bbox prompt, explicitly requesting four box coordinates in the model's coordinate convention. Both retain the same eight-sample, gating and cropping procedure. These results reflect transfer to point-output grounders. †On UI-TARS, neither prompt produces accepted boxes, so 0.73% reflects the official center fallback.

We will incorporate the first-click crop, RegionFocus, and GUI-RC comparisons, together with the compute-budget analysis and corresponding implementation details, into the revised paper before the deadline.

AuthorsOpenReview ↗

Follow-up on Our Response and Additional Experiments

Dear Reviewer Poh1,

Just a friendly reminder as the discussion deadline approaches. We hope our response has addressed your concerns.

We have also added results from three training runs and additional inference comparisons to the appendix, and clarified the crop selection ablation.

We look forward to hearing your thoughts and would be happy to answer any remaining questions. Thank you again for your time!

Thank you for the detailed responses and additional experiments.

I retain some reservations about the strength of the connection between ConClick and ReClick. But overall I'm updating my rating from 2 to 3

Comments on the paper

1 comment
Area_Chair_JMNKOpenReview ↗

Urgent Reminder: Discussion Period Ends November 18

Dear Reviewers,

The discussion period will end on November 18, 2026. Please carefully review the authors' responses and actively engage in discussions.

If you have not yet initiated a discussion, please do so immediately. Reviewers are expected to address the authors' rebuttals, provide constructive follow-up comments, and update their evaluations when appropriate.

Please also ensure that your reviews reflect independent human judgment and that any AI assistance is properly documented in accordance with the review guidelines.

Active participation is an essential reviewer responsibility and is critical to ensuring a rigorous and fair review process.

Thank you for your cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.