- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MissClick, a white-box adversarial attack against GUI grounding models. It introduces two objectives: soft-coordinate displacement for untargeted attacks and place-weighted digit cross-entropy for targeted attacks. Both are optimized using standard PGD.
Critical strengths
- I find the problem relevant to GUI agent security, and I appreciate that the authors consider the numerical meaning of generated coordinates rather than treating them as ordinary text tokens.
- The reported improvements over existing attack baselines are substantial, especially for targeted attacks.
- I appreciate the objective ablations, perturbation budget analysis, and additional experiments on transferability and input preprocessing.
Critical weaknesses
- I am not convinced by the methodological novelty. Coordinate-digit position weighting has already been studied in Phi-Ground [1], including the same relative weights used by MissClick-T. Numerical-aware token losses and analysis have also been explored in [2,3,4]. Although these works focus on model training rather than adversarial attacks, I see MissClick mainly as adapting existing numerical modeling ideas to PGD optimization, rather than introducing a fundamentally new attack method.
- The term "execution-aware" is somewhat overstated. MissClick-U maximizes distance from the correct element's center, while MissClick-T optimizes the digits of a fixed target coordinate. Neither objective directly optimizes whether the click falls inside or outside the relevant element boundary. Element size and shape are not explicitly considered. I therefore think the method models coordinate values more directly than actual GUI execution conditions.
- The evaluation assumes white-box gradient access and full-image perturbations, which may not reflect realistic attacker control over GUI content. Cross-model transfer is weak, particularly for targeted attacks, and simple JPEG compression or Gaussian blur substantially reduces attack success. Although these preprocessing results are non-adaptive, I think they raise important questions about practical robustness. Furthermore, the main evaluation covers only two models and one benchmark, without complete agent workflows.
[1] Phi-Ground Tech Report: Advancing Perception in GUI Grounding. arXiv 2025.
[2] Regress, Don't Guess: A Regression-like Loss on Number Tokens for Language Models. ICML 2025.
[3] Advancing Sequential Numerical Prediction in Autoregressive Models. ACL 2025.
[4] Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv 2026.
Questions for authors
- Could the authors clarify how MissClick-T differs from the coordinate-digit loss reweighting explored in [1]? Beyond applying it to adversarial optimization, what is the main methodological novelty?
- Why not directly optimize whether the predicted click falls inside or outside the target bounding box? Could the authors compare against a region-aware attack objective?
- In Tab.14, I notice that clean accuracy is measured on the full dataset, while ASR is evaluated on the original clean-correct subset. I would like to see the preprocessing-only grounding failure rate on the same subset, since some remaining attack successes may simply reflect errors caused by preprocessing rather than adversarial perturbations. I also think adaptive attacks that account for these transformations would provide a more convincing evaluation.
- This paper only targets OSAtlas and UGround, both of which were fine-tuned from the Qwen2-VL, while lacking evaluations against the latest SOTA models. Could the authors evaluate more advanced models, such as UI-Venus-1.5, GUI-Owl-1.5, Qwen3-VL, or Qwen-3.x (selecting a few of these)?
- Could the authors evaluate more realistic attacks with localized screenshot modifications? I would also like to understand whether the approach generalizes beyond the two evaluated coordinate-generating models.
I appreciate the additional experiments. In light of these, I am willing to raise my score to 3. However, this adjustment is based solely on the extra evaluations involving SOTA models and robustness (contributions rooted in empirical experimentation rather than in theory or conceptual innovation). Regarding the authors' other rebuttals, I remain unconvinced. I must point out the following:
In their response, the authors repeatedly pasted the same content (some of which was clearly intended for other reviewers) and even included certain sections three times (e.g., the results for SOTA models such as Qwen3-VL, GUI-Owl-1.5, and UI-Venus-1.5).
I find the authors' defense of their work's novelty unpersuasive. I have direct research experience with white-box attacks across various domains, and I've never considered white-box PGD attacks to be the most critical issue regarding robustness in CUAs. It seems the authors are attempting to justify the novelty of their work primarily by citing the effectiveness of their proposed method. I do not deny that a straightforward optimization process can, at times, constitute a meaningful methodological contribution. However, these precedents do not prove that MissClick offers a contribution of comparable significance.
The authors argue that "
simplicity is common in the adversarial attack literature and does not by itself diminish the contribution," citing examples from other contexts. Yet, SimBA is a purely black-box attack, and SlowFormer is a universal attack—both of which transcend the inherent limitations of white-box PGD. Other cited works either represent earlier explorations of PGD attacks within their respective fields or introduce entirely new scenarios (e.g., human-in-the-loop). More critically, these examples are drawn from vastly different visual attack domains and span a timeframe from 2019 to 2024 (have the authors considered the relevance of these works within their own specific contexts today)? While employing a simple method is not inherently problematic, one must ask: what substance actually remains in this paper’s proposed approach?At the method level, our contribution lies in the design and controlled evaluation of objectives for click attacks, rather than in a new weighting formula." However, thedesign and controlled evaluationof such objectives have also been explored in existing literature.White-box PGD attacks are inherently more limited in practice compared to black-box attacks, so they should show more novelty to achieve the same level of contribution. I have one question: have you tested the transferability of your white-box attack on SOTA commercial models like the latest Claude or GPT? (Since you cannot obtain gradients from these models)
However, I must emphasize to the AC that, in my view, the overall contribution should be evaluated based on both its practical utility and its novelty. The additional experiments provided in the rebuttal largely address my concerns regarding practical utility—performing quite well within the constraints of a white-box PGD attack scenario—though the work does not transcend those limitations. I urge the AC not to overlook a lack of novelty simply because of the work's practical utility; conversely, one should not disregard its empirical contributions due to a lack of novelty (which is why I have raised my score to 3). In short, while I would not object to this paper being accepted at ICLR (as there are certainly many weaker submissions), I am not sufficiently convinced to actively defend it against the concerns raised by other reviewers.
[1] On the Robustness of GUI Grounding Models Against Image Attacks. CVPR25 workshop
[2] Dissecting Adversarial Robustness of Multimodal LM Agents. ICLR25
[3] MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents. NeurIPS25
[4] Learn where to click from yourself: On-policy self-distillation for gui grounding. arXiv 2026.