GistBackdoor: Intent-Triggered Backdoors in GUI Grounding

GUI agents rely on grounding to locate the interface elements specified by instructions. A backdoor in grounding can redirect an agent's clicks to attacker-chosen elements, turning benign requests into consequential actions such as consenting to tracking, retaining login credentials, or authorizing recurring charges. Existing grounding backdoors, however, rely on identifiable input cues, such as visual patterns, fixed strings, or composite triggers, that offer a basis for detection during use. We introduce GistBackdoor, an intent-triggered backdoor implanted by poisoning community-curated fine-tuning corpora. It redirects instructions expressing a targeted intent to an attacker-chosen interface element across different phrasings, without requiring any additional input cue. Because the same intent also occurs in legitimate requests, filtering instructions by intent risks blocking benign use. The attack manifests in the output as a semantic mismatch between the instruction and the selected element. Across three safety-critical scenarios and three backbones, GistBackdoor achieves average attack success rates of 82.8% on held-out pages and 71.7% on held-out paraphrases, compared with 10.7% and 10.1%, respectively, for the unpoisoned fine-tuning baselines. We compare GistBackdoor with existing backdoor attacks and evaluate representative detectors spanning textual, visual, cross-modal, and representation-based detection. While the compared attacks expose cues targeted by existing detection mechanisms, GistBackdoor remains difficult for the evaluated detectors to identify while maintaining low false-positive rates on benign inputs.