acceptodds
Under review as a conference paper at ICLR 2027

MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models

Abstract

Recent GUI visual grounding models generate screen coordinates as digit-token sequences that are parsed into numerical values and mapped to executable clicks. This generation-to-execution interface creates an attack surface that existing objectives over visual representations or coordinate-token sequences do not explicitly model. Although each coordinate digit is predicted as a token, its spatial effect after parsing depends on decimal position: changing a hundreds-place digit by one shifts the coordinate by 100 units, whereas the same change at the ones place shifts it by one. This mismatch motivates attack objectives that account for both numerical coordinate structure and click execution. Moreover, untargeted and targeted attacks require different objectives because they aim to move the click outside the correct region and into an attacker-specified region, respectively. We propose MissClick, an execution-aware white-box attack that aligns optimization with click-level success conditions. MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted redirection. On OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07% and 72.93% (+16.62 and +30.72 pp), while MissClick-T achieves targeted success rates of 44.86% and 62.67% (+31.73 and +47.06 pp). Among the evaluated objectives, soft-coordinate displacement performs best for untargeted attacks, whereas place-weighted target-digit optimization performs best for targeted attacks, supporting goal-specific execution-aware objective design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, adversarial example, coordinate generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pn5kOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MissClick, a white-box adversarial attack against GUI grounding models. It introduces two objectives: soft-coordinate displacement for untargeted attacks and place-weighted digit cross-entropy for targeted attacks. Both are optimized using standard PGD.

Critical strengths
  • I find the problem relevant to GUI agent security, and I appreciate that the authors consider the numerical meaning of generated coordinates rather than treating them as ordinary text tokens.
  • The reported improvements over existing attack baselines are substantial, especially for targeted attacks.
  • I appreciate the objective ablations, perturbation budget analysis, and additional experiments on transferability and input preprocessing.
Critical weaknesses
  • I am not convinced by the methodological novelty. Coordinate-digit position weighting has already been studied in Phi-Ground [1], including the same relative weights used by MissClick-T. Numerical-aware token losses and analysis have also been explored in [2,3,4]. Although these works focus on model training rather than adversarial attacks, I see MissClick mainly as adapting existing numerical modeling ideas to PGD optimization, rather than introducing a fundamentally new attack method.
  • The term "execution-aware" is somewhat overstated. MissClick-U maximizes distance from the correct element's center, while MissClick-T optimizes the digits of a fixed target coordinate. Neither objective directly optimizes whether the click falls inside or outside the relevant element boundary. Element size and shape are not explicitly considered. I therefore think the method models coordinate values more directly than actual GUI execution conditions.
  • The evaluation assumes white-box gradient access and full-image perturbations, which may not reflect realistic attacker control over GUI content. Cross-model transfer is weak, particularly for targeted attacks, and simple JPEG compression or Gaussian blur substantially reduces attack success. Although these preprocessing results are non-adaptive, I think they raise important questions about practical robustness. Furthermore, the main evaluation covers only two models and one benchmark, without complete agent workflows.

[1] Phi-Ground Tech Report: Advancing Perception in GUI Grounding. arXiv 2025.

[2] Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models. ICML 2025.

[3] Advancing Sequential Numerical Prediction in Autoregressive Models. ACL 2025.

[4] Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv 2026.

Questions for authors
  • Could the authors clarify how MissClick-T differs from the coordinate-digit loss reweighting explored in [1]? Beyond applying it to adversarial optimization, what is the main methodological novelty?
  • Why not directly optimize whether the predicted click falls inside or outside the target bounding box? Could the authors compare against a region-aware attack objective?
  • In Tab.14, I notice that clean accuracy is measured on the full dataset, while ASR is evaluated on the original clean-correct subset. I would like to see the preprocessing-only grounding failure rate on the same subset, since some remaining attack successes may simply reflect errors caused by preprocessing rather than adversarial perturbations. I also think adaptive attacks that account for these transformations would provide a more convincing evaluation.
  • This paper only targets OSAtlas and UGround, both of which were fine-tuned from the Qwen2-VL, while lacking evaluations against the latest SOTA models. Could the authors evaluate more advanced models, such as UI-Venus-1.5, GUI-Owl-1.5, Qwen3-VL, or Qwen-3.x (selecting a few of these)?
  • Could the authors evaluate more realistic attacks with localized screenshot modifications? I would also like to understand whether the approach generalizes beyond the two evaluated coordinate-generating models.
1 comment

I appreciate the additional experiments. In light of these, I am willing to raise my score to 3. However, this adjustment is based solely on the extra evaluations involving SOTA models and robustness (contributions rooted in empirical experimentation rather than in theory or conceptual innovation). Regarding the authors' other rebuttals, I remain unconvinced. I must point out the following:

  1. In their response, the authors repeatedly pasted the same content (some of which was clearly intended for other reviewers) and even included certain sections three times (e.g., the results for SOTA models such as Qwen3-VL, GUI-Owl-1.5, and UI-Venus-1.5).

  2. I find the authors' defense of their work's novelty unpersuasive. I have direct research experience with white-box attacks across various domains, and I've never considered white-box PGD attacks to be the most critical issue regarding robustness in CUAs. It seems the authors are attempting to justify the novelty of their work primarily by citing the effectiveness of their proposed method. I do not deny that a straightforward optimization process can, at times, constitute a meaningful methodological contribution. However, these precedents do not prove that MissClick offers a contribution of comparable significance.

The authors argue that "simplicity is common in the adversarial attack literature and does not by itself diminish the contribution," citing examples from other contexts. Yet, SimBA is a purely black-box attack, and SlowFormer is a universal attack—both of which transcend the inherent limitations of white-box PGD. Other cited works either represent earlier explorations of PGD attacks within their respective fields or introduce entirely new scenarios (e.g., human-in-the-loop). More critically, these examples are drawn from vastly different visual attack domains and span a timeframe from 2019 to 2024 (have the authors considered the relevance of these works within their own specific contexts today)? While employing a simple method is not inherently problematic, one must ask: what substance actually remains in this paper’s proposed approach?

  • In terms of the problem setting, the field of CUA PGD attacks is already quite crowded [1-3], and these are works from as far back as two years ago.
  • Regarding the methodology, the authors provide a table to justify the novelty of their "coordinate-digit loss." However, their defense essentially boils down to one point: "Our setting differs from theirs; we focus on attacks and defenses rather than training." Notably, work like GUI-SD [4] not only proposes a method but also provides an empirical analysis of how weighting individual digital tokens affects model performance. Furthermore, this "coordinate-digit loss" is not even presented as a core innovation in those related studies. The authors claim, "At the method level, our contribution lies in the design and controlled evaluation of objectives for click attacks, rather than in a new weighting formula." However, the design and controlled evaluation of such objectives have also been explored in existing literature.

White-box PGD attacks are inherently more limited in practice compared to black-box attacks, so they should show more novelty to achieve the same level of contribution. I have one question: have you tested the transferability of your white-box attack on SOTA commercial models like the latest Claude or GPT? (Since you cannot obtain gradients from these models)

However, I must emphasize to the AC that, in my view, the overall contribution should be evaluated based on both its practical utility and its novelty. The additional experiments provided in the rebuttal largely address my concerns regarding practical utility—performing quite well within the constraints of a white-box PGD attack scenario—though the work does not transcend those limitations. I urge the AC not to overlook a lack of novelty simply because of the work's practical utility; conversely, one should not disregard its empirical contributions due to a lack of novelty (which is why I have raised my score to 3). In short, while I would not object to this paper being accepted at ICLR (as there are certainly many weaker submissions), I am not sufficiently convinced to actively defend it against the concerns raised by other reviewers.

[1] On the Robustness of GUI Grounding Models Against Image Attacks. CVPR25 workshop

[2] Dissecting Adversarial Robustness of Multimodal LM Agents. ICLR25

[3] MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents. NeurIPS25

[4] Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv 2026.

Reviewer_DV5JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MissClick studies white-box image attacks against GUI grounding models that generate coordinates as digit tokens. It uses soft-coordinate displacement for untargeted attacks and place-weighted digit cross-entropy for targeted attacks. On OS-Atlas and UGround with ScreenSpot-v2, both objectives substantially improve attack success over the evaluated baselines. The paper also examines digit-level behavior, transferability, and sensitivity to input preprocessing.

Critical strengths
  • The comparison between untargeted and targeted objectives is useful. Soft-coordinate distance works best for disruption, while weighted digit CE works best for redirection. The observation that better high-order digit accuracy can outweigh worse low-order accuracy gives some insight beyond the aggregate attack rates.
Critical weaknesses
  • I am not yet convinced that the contribution is sufficiently distinguished from existing numerical objectives. Phi-Ground already considers the same relative digit weights, and numerical expectations appear in prior losses. Moreover, much of the targeted improvement comes from switching from Token CE to Digit CE; place weighting adds a smaller gain. The paper needs a clearer separation of these effects and a closer comparison with existing numerical losses.
  • The practical relevance of the attack remains unclear. It assumes access to model gradients and the ability to optimize the entire screenshot for each instruction, whereas an attacker may control only a small region of the rendered interface. Targeted transfer is low, and simple JPEG compression or blurring sharply reduces attack success. These results do not invalidate the white-box findings, but the paper would be more compelling with a concrete deployment scenario where its assumptions hold, or evidence under more restricted input control.
Questions for authors
  • Does Digit CE change both the supervised positions and the softmax vocabulary relative to Token CE? If so, do you have results separating these changes? This affects how much of the reported gain can be attributed to the proposed numerical structure.
  • What deployment scenario gives an attacker access to model gradients and control over the entire screenshot? Given the low targeted transfer and sensitivity to preprocessing, do you have evidence under restricted input control or preprocessing-aware optimization? Such evidence would strengthen the practical relevance of the attack.
Reviewer_yMKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MissClick, an execution-aware adversarial attack against coordinate-generating GUI grounding models. It introduces an untargeted objective that maximizes soft-click displacement and a targeted objective based on place-weighted digit cross-entropy. The method applies projected gradient updates to the input screenshot while keeping the victim model fixed. Experiments on OS-Atlas and UGround using ScreenSpot-v2 show substantial improvements over several baselines.

Critical strengths
  1. The paper studies a relevant security problem and usefully shifts the attack objective from coordinate-token prediction to the final executed click.
  2. The distinction between untargeted and targeted attacks is intuitive, and the use of decimal place values is well motivated by their different effects on coordinate displacement.
  3. The evaluation covers two models, three GUI platforms, and several baselines, with additional analyses of perturbation budgets, transferability, and input preprocessing.
Critical weaknesses
  1. The threat model does not fully specify the attacker's required information. Although the paper assumes white-box access to the model, the actual optimization also requires ground-truth coordinates for MissClick-U and an annotated target region for MissClick-T. In particular, target regions are sampled from annotated ScreenSpot-v2 elements. Access to these annotations is an additional oracle assumption, not implied by access to model weights. The paper also assumes that the attacker can directly modify the screenshot supplied to the model, but does not explain how this access would be obtained in a realistic GUI pipeline. The current evaluation therefore demonstrates an annotation-assisted white-box attack on benchmark screenshots. The authors should clarify these capabilities and discuss how source and target coordinates could be obtained in practice.

  2. The digit-restricted probabilities differ from those used by the decoder. Equation 6 normalizes logits over only the ten digit tokens, while final generation uses the full vocabulary. A target digit can have the highest probability among digits without being the full-vocabulary argmax. The authors should clarify this distinction and examine how often the digit-restricted prediction agrees with the actual greedy token.

  3. The targeted loss is not fully aligned with the region-level success criterion. For OS-Atlas, MissClick-T optimizes all four coordinates of the target bounding box, although success only requires the executed click to fall inside the target region. It is unclear whether the gain comes from place-weighting itself or from providing a more specific coordinate target. A comparison with a center-coordinate or region-aware objective would help isolate the contribution.

Questions for authors
  1. Does the threat model assume access to exact source and target annotations? If so, how could an attacker obtain them outside a benchmark setting?
  2. How sensitive is the attack to approximate or noisy source and target coordinates?
  3. How often do digit-restricted argmax predictions agree with full-vocabulary greedy generation, and how does prefix divergence affect attack success?
3 comments

Comment B - W2: Clarifying the distinction between digit-restricted probabilities and full-vocabulary decoding (1/2)

Review: The digit-restricted probabilities differ from those used by the decoder. Equation 6 normalizes logits over only the ten digit tokens, while final generation uses the full vocabulary. A target digit can have the highest probability among digits without being the full-vocabulary argmax. The authors should clarify this distinction and examine how often the digit-restricted prediction agrees with the actual greedy token.

We thank the reviewer for the opportunity to clarify. We first clarify the distinction between the two distributions, then explain the role of Eq.6 in the original formulation and our refinement of MissClick-T. Finally, we provide additional experimental results to address the reviewer's concern.

1. Clarifying the distinction.

Both the digit-restricted distribution and the full-vocabulary distribution are derived from the same logits at the same prediction position; they differ in their normalization domain and in when they are used. The generation stage does not use the digit-restricted distribution: generation applies a softmax over the full vocabulary and selects the next token by full-vocabulary greedy decoding, which is the model’s standard generation protocol, used uniformly in all evaluations and unchanged by our attack. The digit-restricted distribution is defined by Eq.6 and is used only during attack optimization, where it constructs the attack loss; the final click is generated through the model’s usual full-vocabulary decoding. Mathematically, Eq.6 is a re-normalization of the full-vocabulary distribution over the digit set. The re-normalization factor does not change the relative ordering among digits, so the digit-restricted argmax is the same as the most likely digit under the full-vocabulary distribution. However, the full-vocabulary argmax may fall on a non-digit token, so the two argmaxes may indeed disagree.

2. Role of Eq.6 in the submitted formulation.

In the submitted formulation, MissClick-U and MissClick-T share the digit-restricted distribution defined in Eq.6.

For MissClick-U, the objective is built on a soft coordinate, d~k,j=∑d∈Dd⋅pk,j(d)\tilde d_{k,j}=\sum_{d\in\mathcal{D}}d\cdot p_{k,j}(d). This is a conditional expectation over digit values: it asks what numerical value the model expects at this position, given that the position carries a digit. The expectation is defined only over tokens with a numerical value; separators, brackets, and terminators have no corresponding decimal value and cannot enter the sum. The digit set is therefore the natural support for a differentiable numerical expectation, and Eq.6 provides exactly the conditional distribution needed for this expectation.

For MissClick-T (submitted), the implementation reused the digit-restricted distribution. Its cross-entropy loss optimizes the target digit’s probability among the ten digits and does not directly account for non-digit tokens.

3. Refinement of the targeted objective.

Motivated by the reviewer's observation, we revisited the targeted objective and further distinguished the probability distributions used by MissClick-U and MissClick-T.

For MissClick-U, we retain Eq.6 to construct the differentiable numerical expectation.

For MissClick-T, we revise the objective to use place-weighted full-vocabulary cross-entropy over the full target sequence. Coordinate-digit positions retain their decimal place weights; non-digit positions have weight 1. With full-vocabulary normalization, the target digit’s probability also depends on the logits of non-digit tokens. This better reflects the token selection used in greedy decoding.

The revision retains the decimal place weights and changes both the normalization domain and the supervised positions relative to the submitted objective. Section 5 then separates the effects of the two changes.

Comment F - Q3: Argmax agreement and prefix divergence

Review: How often do digit-restricted argmax predictions agree with full-vocabulary greedy generation, and how does prefix divergence affect attack success?

Response: We thank the reviewer for these questions. We answer them in turn.

1. How often the two predictions agree.

To address the reviewer’s question about the actual greedy token, we conduct an additional analysis on the adversarial examples generated by MissClick-U and the submitted MissClick-T.

At each evaluated coordinate-digit prediction position tt, we obtain the model logits ztz_t under the same prediction context and compare two predictions:

dt∗=arg⁡max⁡d∈Dzt(d),vt∗=arg⁡max⁡v∈Vzt(v),d_t^{\ast}=\arg\max_{d\in\mathcal{D}}z_t(d),\qquad v_t^{\ast}=\arg\max_{v\in\mathcal{V}}z_t(v),

where dt∗d_t^{\ast} is the digit-restricted argmax over the ten digit tokens, and vt∗v_t^{\ast} is the full-vocabulary argmax under the same prediction context. A position is counted as agreement if the two predictions select the same token. If the full-vocabulary argmax selects a non-digit token at an evaluated coordinate-digit position, the position is counted as disagreement.

The agreement rate is

Ragree=1Ndigit∑t=1NdigitI[dt∗=vt∗],R_{\mathrm{agree}}=\frac{1}{N_{\mathrm{digit}}}\sum_{t=1}^{N_{\mathrm{digit}}}\mathbb{I}\left[d_t^{\ast}=v_t^{\ast}\right], where NdigitN_{\mathrm{digit}} is the total number of evaluated coordinate-digit positions across all samples.

ModelAttackAgreementAgreement rateFull-vocabulary top-1 at disagreements
OS-AtlasMissClick-T (submitted)8438 / 866297.41%,, ),(
UGroundMissClick-T (submitted)4528 / 461898.05%,, )
OS-AtlasMissClick-U11777 / 1202597.94%,, ),(, )
UGroundMissClick-U6303 / 647497.36%,, )

The agreement rates range from 97.36% to 98.05% across the four settings. These results show that digit-restricted and full-vocabulary top-1 predictions agree at most evaluated coordinate-digit positions, but the disagreement identified by the reviewer does occur.

We further inspected the full-vocabulary top-1 predictions at disagreement positions. The observed competing tokens were coordinate-format delimiters, including ,, ),(, and ).

2. How prefix divergence affects attack success.

For each evaluated sample, we compare the coordinate digits in the reference sequence used during optimization with those in the actual greedy output. A sample is counted as divergent if any coordinate digit differs.

ASR (all) is the fraction of all evaluated samples whose final click falls inside the target region. ASR (divergent) applies the same success criterion only to samples whose generated coordinate digits differ from the reference sequence.

ModelTotal samplesDivergent samplesASR (divergent)ASR (all)
OS-Atlas769766 (99.6%)342/766 (44.6%)345/769 (44.9%)
UGround801746 (93.1%)447/746 (59.9%)502/801 (62.7%)

Coordinate-digit divergence is common, and attacks can still succeed when it occurs: ASR among divergent samples is 44.6% on OS-Atlas and 59.9% on UGround. This shows that a mismatch in coordinate digits does not necessarily mean the final click misses the target region.

When the generated prefix differs from the reference prefix, the model may make different predictions for the following coordinate tokens, so the predictions favored under the reference prefix may not fully carry over to actual generation. With other digits unchanged, the same digit change at a higher decimal place produces a larger coordinate shift and can move the final click outside the target region; divergence at a lower decimal place changes the coordinate by a small amount. A divergence does not by itself mean failure: the attack still succeeds if the final click falls inside the target region. Our reported attack success rates are computed from the final clicks produced by actual generation.

We sincerely thank the reviewer for their time and effort in evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.