acceptodds
Under review as a conference paper at ICLR 2027

GistBackdoor: Intent-Triggered Backdoors in GUI Grounding

Abstract

GUI agents rely on grounding to locate the interface elements specified by instructions. A backdoor in grounding can redirect an agent's clicks to attacker-chosen elements, turning benign requests into consequential actions such as consenting to tracking, retaining login credentials, or authorizing recurring charges. Existing grounding backdoors, however, rely on identifiable input cues, such as visual patterns, fixed strings, or composite triggers, that offer a basis for detection during use. We introduce GistBackdoor, an intent-triggered backdoor implanted by poisoning community-curated fine-tuning corpora. It redirects instructions expressing a targeted intent to an attacker-chosen interface element across different phrasings, without requiring any additional input cue. Because the same intent also occurs in legitimate requests, filtering instructions by intent risks blocking benign use. The attack manifests in the output as a semantic mismatch between the instruction and the selected element. Across three safety-critical scenarios and three backbones, GistBackdoor achieves average attack success rates of 82.8% on held-out pages and 71.7% on held-out paraphrases, compared with 10.7% and 10.1%, respectively, for the unpoisoned fine-tuning baselines. We compare GistBackdoor with existing backdoor attacks and evaluate representative detectors spanning textual, visual, cross-modal, and representation-based detection. While the compared attacks expose cues targeted by existing detection mechanisms, GistBackdoor remains difficult for the evaluated detectors to identify while maintaining low false-positive rates on benign inputs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Backdoor attack, GUI grounding, Intent triggers, Data poisoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dm2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GistBackdoor, which poisons GUI grounding fine-tuning data to associate naturally expressed user intents with attacker-chosen interface elements, without adding textual markers or modifying screenshots at inference time. The authors evaluate attack effectiveness, generalization across pages and phrasings, preservation of benign utility, and detection performance across three backbones and three real-webpage scenarios. Further discussion would be useful regarding the practical feasibility of injecting 5% poisoned data plus an additional 5% cover data in the main experiments, and whether user intent remains present in the instructions passed from the planner to the grounder.

Critical strengths

The paper investigates a valuable question: replacing explicit textual or visual triggers with naturally expressed user intent, without requiring additional cues at inference time. This offers an interesting perspective on supply-chain risks in GUI agents. The experiments include several careful design choices, including host-disjoint training and evaluation pages, SFT and SFT+cover controls to isolate the contribution of cover data, and reproductions of multiple baseline attacks under a shared configuration. Poisoning-rate sweeps, layer-block ablations, and linear probes provide additional analysis. Attack success rates increase substantially across all three backbones and three scenarios, while general grounding utility is largely preserved. The authors also discuss limitations transparently and provide a thorough ethics statement.

Critical weaknesses
  1. The evidence for stealth remains insufficient. Most evaluated detectors target traces associated with explicit triggers, such as rare strings, syntactic templates, and image patches. Since GistBackdoor avoids these traces by design, the all-cross GistBackdoor row provides limited support for its stealth. The paper itself identifies semantic inconsistency between instruction intent and the selected element as a remaining signal, but leaves inference-time instruction–output consistency checking to future work. The claim that the attack is difficult to detect therefore lacks sufficient evaluation against a defense directly targeting this signal. The small judges in Appendix A.4 flag almost none of the poisoned samples while flagging many legitimate samples, with a false-positive rate reaching 66.5%. These anomalous results raise concerns about the reliability of the current semantic audit and do not establish that semantic auditing is ineffective. Evaluation with a stronger LLM/VLM judge would strengthen this conclusion.

  2. The evaluation of generalization across phrasings remains limited. Each scenario uses only seven held-out paraphrases, written by the same authors who constructed the training expressions. Although the results demonstrate generalization to unseen phrasings, they do not sufficiently establish applicability to a broader range of natural expressions. Expanding the test set and incorporating expressions from independent authors or real users would provide stronger evidence of intent-trigger generalization.

  3. The interpretation of attack comparisons needs a clearer scope. The ASRpage and ASRpara columns in Table 3(a) evaluate all attacks using GistBackdoor’s intent instructions, without supplying the triggers required by BadNets, VisualTrap, and CBA. Their low scores in these columns are therefore expected and do not indicate that these attacks are generally weaker. Although the “Own trigger” column reports performance under each attack’s activation conditions, the two ASR columns primarily show that fixed-trigger attacks do not generalize to these intent expressions; they do not establish the general superiority of GistBackdoor. Additional comparisons with semantic or topic-conditioned backdoors, beyond the existing syntactic HiddenKiller baseline, would help clarify the novelty boundary.

Questions for authors

Please see the weakness.

4 comments
AuthorsOpenReview ↗

Response part 1

Thank you for the constructive assessment and for identifying the limits of the original stealth, expression-generalization and attack-comparison evidence. We address these points with direct semantic checks, independent expression testing and a clearer account of task adaptation and activation conditions.

Strong instruction–action consistency checking (W1).

We agree that the explicit-trigger detector matrix does not establish resistance to semantic auditing. The semantic-audit results reported in Appendix A.4 require correction and do not support the original broad stealth interpretation. We will replace the affected evidence with the direct consistency evaluation below.

To address the requested stronger defense, we evaluate DeepSeek Flash, Kimi K3 and GLM-5.3-Flash visual judges, directly checking whether the selected control satisfies the instruction. Each checker receives the user instruction, the screenshot with the predicted click marked, and a local crop, without the correct target or attack labels. We evaluate 300 requests (150 target-intent, 75 neighboring-intent and 75 other legitimate requests), checking both poisoned-model and SFT predictions: 600 actions per checker. The reference labels contain 213 noncompliant, 319 compliant and 68 unassessable actions; TPR/FPR use the first two groups. The common policy blocks inconsistent or uncertain judgments. Accepted ASR counts payload hits that pass the check among all 150 target-intent requests; the unchecked ASR on this subset is 96.0%. This is a separate evaluation from the 740-case independent-expression test.

API visual judgeBlocking TPRFalse-blocking FPRAccepted payload-hit ASRCheck latency p50 / p95Estimated USD / 1,000 checks
DeepSeek Flash94.4% (201/213)7.8% (25/319)1.3% (2/150)1.18 / 14.22 s$0.23
Kimi K393.0% (198/213)4.4% (14/319)2.7% (4/150)4.65 / 10.35 s$3.31
GLM-5.3-Flash93.4% (199/213)6.3% (20/319)4.0% (6/150)2.41 / 4.83 s$0.26

TPR/FPR reflect the blocking policy, including insufficient-evidence decisions.

API costs are estimated from measured token usage and listed prices per 1,000 checks. Price references dated November 10, 2026: DeepSeek, Kimi, GLM. Latencies measure judge calls.

Strong, contextualized semantic checking is effective in this setting. It reduces accepted ASR from 96.0% to 1.3–4.0%, while falsely blocking 4.4–7.8% of compliant actions. The additional judge call has median latency 1.18–4.65 s, compared with 0.806 s for the 2B grounder in a small benchmark of 54 observations. These measurements describe effectiveness, false blocking and added inference cost. Blocking stops actions without replanning, so accepted ASR and successful task completion remain different outcomes.

As a complementary evaluation for compact deployments, we test local 4B and 8B visual checkers on the same actions:

Local visual checkerNoncompliant blocking, TPRCompliant false blocking, FPRAccepted payload-hit ASR
Qwen3-VL-4B58.7% (125/213)12.5% (40/319)96.0% → 37.3% (56/150)
Qwen3-VL-8B81.2% (173/213)11.3% (36/319)96.0% → 9.3% (14/150)

Both local checkers mitigate the attack, with the 8B checker substantially stronger, but accepted ASR remains 37.3% and 9.3%, compared with 4.0% and 2.0% for checked SFT outputs. This local result complements the strong-model evidence; it does not support a blanket claim of resistance to semantic checking.

We will revise the stealth wording in the abstract and introduction, replace the affected Appendix A.4 evidence, and add direct consistency measurements to Detection Evaluation. Table 4 will describe the methods and criteria actually tested. The conclusion and Limitations will recognize contextualized semantic checking as an effective evaluated defense, with its measured false blocking and inference overhead.

AuthorsOpenReview ↗

Response part 2

Independent expression coverage (W2).

We agree that seven expressions written by the training-data authors provide limited evidence for broader expression coverage. We expanded COOKIE testing to 180 same-intent instructions from three independent model sources—GLM, DeepSeek and Kimi, 60 each. Generators received neither the training instruction bank, the original seven expressions nor grounding predictions. Duplicate/overlap filtering and intent-consistency checks by a model other than each expression's generator were applied. These are independently model-generated expressions, rather than independently collected human-user instructions.

The evaluation comprises 740 matched instruction–page cases across 30 pages. We report each source separately to show whether the effect depends on a particular generator:

Independent sourceInstructionsPoisoned ASRSFT payload-hit rateIncrease, pp [95% CI]
GLM6096.3%10.8%+85.6 [76.5, 94.8]
DeepSeek6096.9%7.7%+89.2 [83.2, 94.9]
Kimi6096.7%9.6%+87.1 [78.6, 95.5]
All sources18096.6%9.3%+87.3 [80.1, 94.5]

Rates are averaged equally over instructions; confidence intervals use paired page-cluster bootstrap. Each source contributes 60 expressions. Increases are computed from unrounded rates. These are comparisons on matched poisoned-model/SFT inputs.

The poisoned model reaches 96.3–96.9% ASR across all three sources, while SFT remains at 7.7–10.8%. Thus, the added evidence extends beyond a small author-written phrase set and beyond one generation source.

We also examine less training-similar wording using Qwen3-Embedding-0.6B and the 504 unique expressions used in poison training. The complete exploratory mean-distance grouping is:

Distance groupMean-distance rangeInstructionsPoisoned ASRSFT payload-hit rateIncrease, pp [95% CI]
G10.222–0.2643497.8%6.9%+90.9 [83.3, 97.9]
G20.266–0.2832397.4%12.6%+84.8 [74.2, 94.8]
G30.286–0.3204197.7%6.1%+91.5 [85.6, 97.9]
G40.322–0.3562998.4%13.3%+85.1 [73.5, 95.7]
G50.359–0.4072995.0%12.0%+83.0 [70.8, 94.5]
G60.411–0.5412492.3%7.2%+85.1 [77.4, 92.7]

Rates are averaged equally across instructions; intervals use paired page-cluster bootstrap, and increases use unrounded rates. Distance stratification is exploratory.

The most distant group contains 24 expressions, with mean distances 0.411–0.541, and retains 92.3% ASR versus 7.2% for SFT—an increase of 85.1 points [95% CI: 77.4–92.7]. To additionally check similarity to a single training phrase, the farthest nearest-expression group contains 49 expressions, with nearest distances 0.206–0.400; ASR is 93.7% versus 12.3%, an increase of 81.3 points [71.3–91.7].

We will add the source-wise and distance-stratified results to Main Results. They support within-intent expression transfer on COOKIE, including less training-similar expressions, while representative human-user language and additional intent–payload pairs remain outside this new evaluation. Embedding distance is a similarity proxy, and the distant-group results define neither a universal activation cutoff nor a strictly monotonic distance–ASR relationship.

AuthorsOpenReview ↗

Response part 3

Comparison scope and task-adapted semantic reference (W3).

We agree that the low ASRpage/ASRpara values for explicit-trigger baselines do not establish general superiority. These columns evaluate unmodified intent instructions, where several baselines' required triggers are absent. Their low activation is therefore expected. The Own-trigger column instead evaluates each attack under its required activation conditions. We will separate these two questions in Table 3(a), its caption and the accompanying discussion.

For the task comparison, the available methods do not directly match the full intent-triggered, native-control setting. We therefore use task-adapted versions of representative attacks. We retain their characteristic trigger mechanisms and express their supervision in the grounding model’s coordinate format, using a common backbone and matched poisoning budget and fine-tuning settings.

MethodAdaptation to grounding
BadNetsAppend the textual trigger to a grounding instruction and supervise the attacker-selected control’s coordinates.
HiddenKillerRewrite grounding instructions into the trigger syntactic pattern and supervise the payload control’s coordinates.
VisualTrapApply the visual trigger to screenshots and retain the trigger patch’s location as the attack target.
CBAPlace the composite textual keys within a single grounding instruction, with coordinate supervision for the payload.

VPI becomes comparable through the same task-adaptation process. We instantiate its semantic trigger scenario as requests to decline cookies and its desired response as selecting the page’s Accept control. In this verifiable setting, the known payload box already specifies the desired coordinate response, making teacher-generated replies unnecessary. VPI permits human annotators as well as a teacher model to provide manipulated responses (Yan et al., 2024, §3). Pairing the original instruction and screenshot with that coordinate implements its semantic-poisoning construction in grounding.

For the basic adaptation without additional task-specific benign cover, the resulting recipe coincides with our existing no-cover ablation: 19,000 general clean grounding examples plus 1,000 poisoned intent–coordinate pairs. We will identify this existing condition as a VPI-style grounding adaptation using known target coordinates, making the recipe correspondence explicit. This is a basic adaptation without extra scenario-specific cover; VPI itself is not defined by the absence of such cover. Comparing it with the cover-augmented recipe examines the effect of task-specific benign supervision within the shared semantic-poisoning framework.

This no-cover reference is distinct from the “neither N nor U” control in the N/U ablation, which retains the common benign cover.

These changes clarify the contribution boundary: the semantic-poisoning principle is shared with prior work, while our experiments characterize its use for intent-triggered GUI actions. The no-cover correspondence supplies a task-specific semantic reference; differences in activation conditions remain separate from effectiveness under each method's own trigger.

Reviewer_Dm2mOpenReview ↗

Official Comment by Reviewer

Thank you for the rebuttal. It has resolved my concern. I will maintain my score.

Reviewer_fsXZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GistBackdoor, an intent-triggered data poisoning attack against GUI grounding models. Unlike conventional GUI backdoors that rely on explicit textual or visual triggers, GistBackdoor uses the semantic intent of legitimate user instructions to redirect grounding predictions to attacker-chosen interface elements. The attack is implemented by poisoning instruction–coordinate pairs in fine-tuning data, without modifying inference-time inputs or the training procedure.

The authors evaluate the attack across three safety-critical scenarios (COOKIE, LOGIN, and DONATION) and three grounding backbones. The results show average attack success rates of 82.8% on unseen pages and 71.7% on held-out instruction paraphrases, while largely preserving benign grounding performance. The paper further compares GistBackdoor with existing backdoor attacks, investigates poisoning rates and benign cover samples, and evaluates six representative detection methods, none of which successfully detects GistBackdoor under the evaluated criteria.

Critical strengths

S1. A practically relevant intent-triggered backdoor attack on GUI grounding.

Unlike conventional GUI backdoor attacks that rely on explicit textual or visual triggers, this paper explores an intent-triggered attack that redirects grounding predictions based on the semantic intent of legitimate user instructions, without modifying inference-time inputs. Experiments across three safety-critical scenarios and three backbones demonstrate its effectiveness on unseen web pages and held-out instruction paraphrases, while largely preserving benign grounding performance. These results highlight a meaningful security risk in GUI grounding models.

Critical weaknesses

W1. Limited technical novelty and generalizability.

The proposed method mainly relies on heuristic Intent–Payload selection, paraphrase generation, and standard data poisoning, without introducing a new optimization algorithm or a principled trigger selection strategy. Moreover, the evaluation only covers three manually selected Intent–Payload pairs, leaving its generalizability to broader GUI scenarios unclear.

W2. Insufficient analysis of over-triggering and negative sampling.

Although the authors introduce near-domain and unrelated-instruction negatives to reduce unintended activation, their individual effects on over-triggering are not evaluated. More importantly, some legitimate instructions still exhibit high over-trigger rates (21.5% for "Manage my cookie settings" and 19.8% for "Show the cookie details"), raising concerns about the attack's selectivity and stealthiness.

W3. Insufficient evaluation of poisoned training data detectability.

The main experiment injects 1,000 poisoned samples into a 20,000-sample training corpus. These samples repeatedly associate the intent to decline tracking with the "Accept all" element, potentially creating detectable semantic inconsistencies. Although the authors evaluate six detectors and conduct a small-scale semantic audit, they do not evaluate whether poisoned samples can be identified within the actual contaminated training corpus. This leaves the claimed stealthiness of the poisoning insufficiently supported.

W4. The technical novelty is further weakened by VPI (Yan et al., 2024), which has already explored semantic-conditioned backdoors through instruction-tuning data poisoning without requiring explicit inference-time triggers. The paper does not sufficiently clarify its methodological distinction from VPI beyond adapting the attack to GUI grounding.

Questions for authors

Q1. How are effective Intent–Payload pairs selected beyond the proposed heuristic? Have the authors evaluated alternative pairs to demonstrate generalizability?

Q2. How do near-domain and unrelated-instruction negatives individually affect ASR and over-trigger rates? Given the approximately 20% over-trigger rates on some legitimate instructions, how do the authors justify the attack's selectivity and stealthiness?

Q3. Can the authors evaluate poisoned-sample detection on the actual 20,000-sample training corpus, rather than separately constructed evaluation sets? In particular, can semantic consistency or corpus-level analysis identify the 1,000 poisoned Instruction–Payload associations?

Q4. Could the authors clarify how the selection and construction of trigger instructions in GistBackdoor differ from those in VPI (Yan et al., 2024), which also leverages semantic conditions for backdoor activation? Simply categorizing VPI as a textual-trigger approach does not adequately distinguish the proposed method from this closely related work.

4 comments
AuthorsOpenReview ↗

Response part 1

Thank you for the questions about novelty, scenario selection, selectivity and corpus-level detectability. We clarify our relationship to semantic-backdoor research, add separate N/U ablations, and evaluate filtering on the actual contaminated training corpus.

Contribution, VPI and attack-pair selection (W1/Q1 and W4/Q4).

We agree that semantic backdoors have an established history, from naturally occurring visual attributes (Bagdasaryan et al., 2020) to semantic-conditioned instruction poisoning. VPI is a semantic-conditioned backdoor attack, and our classification of it as an explicit textual-trigger approach was inaccurate. We will correct Related Work and the corresponding comparison (Yan et al., 2024).

To our knowledge, ours is the first study of semantic backdoors in GUI grounding triggered by the action intent of ordinary, unmodified instructions. Earlier GUI grounding attacks such as VisualTrap use visual triggers. Our contribution concerns this attack setting, its grounding-specific construction and empirical characterization. Training uses standard fine-tuning.

The adaptation has three components. First, the attacker identifies a harmful native control and a conflicting legitimate user intent with a correct alternative on the same page. For COOKIE, declining tracking is the trigger intent and Accept All is the payload; opening preferences is a different action intent despite sharing the topic. Trigger expressions vary the wording of the defined goal. Second, poisoned samples pair the original screenshot and instruction with that page's payload coordinates, allowing its location and surrounding layout to vary. Third, benign grounding supervision and evaluation of neighboring-intent activation, ordinary utility and attack success characterize the association's transfer and selectivity.

VPI's semantic-scenario formulation can already encompass action intents. With a known payload box, the manipulated response can be supplied directly as coordinates, making teacher-generated responses unnecessary for this grounding adaptation. The shared principle remains semantic poisoning; our empirical contribution concerns localization, expression transfer, selectivity and relevant defenses in GUI grounding.

Scenario selection starts from the attacker's desired outcome and the native control that produces it, then identifies a conflicting legitimate intent with an available correct alternative. This rule yields the following concrete GUI constructions:

GUI scenarioTriggering intent → legitimate targetPayloadSecurity consequence
Recycle binRestore selected items → RestoreDelete permanentlyRisk of irreversible data loss
Content editorSave content as a draft → Save draftPublishPremature disclosure of unfinished or private content
Location permissionPrevent website location access → Don't allowAllowLocation access contrary to the user's request
Subscription managementStop charges from the next billing cycle → Turn off auto-renewal / Confirm cancellationKeep subscription / Continue renewalContinued recurring charges

These examples assume that the legitimate and payload controls are available in the same screenshot; subsequent confirmation follows the application's workflow. They illustrate construction feasibility rather than additional experimentally established attack results. The new quantitative experiments in this rebuttal are limited to COOKIE, and we will distinguish the selection rule and these extensions from empirical coverage of evaluated pairs.

AuthorsOpenReview ↗

Response part 2

Independent N/U effects and over-triggering (W2/Q2).

We agree that the combined benign-data ablation did not establish the individual roles of N and U, and that the reported neighboring-request failures reveal imperfect selectivity. We therefore added a four-way COOKIE ablation, using audited, page-appropriate U examples. N covers neighboring cookie-management intents; U covers unrelated legitimate site tasks on cookie-containing pages. We measure both task accuracy and over-triggering—the latter means selecting Accept-all when the request requires another control—and include general GUI accuracy on ScreenSpot-2 (SS2) and ScreenSpot-Pro (SSP).

Negative supervisionPage-test ASRN-task accuracyN over-trigger rateU-task accuracySS2 accuracySSP accuracy
Neither N nor U96.41%77.03%16.22%67.05%89.32%49.42%
N only91.86%91.89%1.35%66.48%89.49%47.97%
U only97.19%74.32%14.86%67.05%88.97%47.97%
N+U98.45%90.54%2.70%65.91%88.46%43.77%

All four models use the same poisoned examples, 5% poisoning, a fixed total training size and the same common benign cover; N and U contain 250 and 172 examples when included. Values are page-macro averages. ASR uses 1,032 page-test cases; N/U task metrics use 72 audited neighboring requests and 173 audited unrelated requests. U-task over-triggering is 0% in every model, although other grounding errors remain.

N improves neighboring-task selectivity in both U backgrounds. Without U, adding N raises N-task accuracy from 77.03% to 91.89% and reduces over-triggering from 16.22% to 1.35%, while ASR decreases from 96.41% to 91.86%. With U present, adding N likewise raises N-task accuracy from 74.32% to 90.54% and reduces over-triggering from 14.86% to 2.70%, while retaining high ASR. The accompanying ASR change therefore depends on the recipe. This selectivity benefit also differs from general GUI utility: N alone leaves SS2 nearly unchanged but lowers SSP accuracy from 49.42% to 47.97%.

U does not show the intended benign-accuracy benefit at this dose. U-task accuracy remains 67.05% without N and changes from 66.48% to 65.91% with N. Adding U to N raises ASR from 91.86% to 98.45%, but also raises the neighboring over-trigger point estimate from 1.35% to 2.70% and lowers SS2/SSP accuracy from 89.49%/47.97% to 88.46%/43.77%. Thus, the observed selectivity improvement is supported by N, while this U block changes the attack–utility balance without demonstrating the expected accuracy preservation. We will revise the interpretation of benign supervision accordingly.

AuthorsOpenReview ↗

Response part 3

False activation outside the intended trigger condition is a recognized issue in backdoor poisoning. For example, CBA (Huang et al., 2024) explicitly evaluates false-triggered rates and uses negative examples to improve trigger specificity. Our high-ASR setting prioritizes reliable attack activation, and the neighboring-request failures expose a selectivity cost of that operating point. We therefore also examined the attack-success–over-triggering trade-off across poisoning rates:

Poisoning ratePage-test ASRNeighboring over-trigger rate (214 requests)
Approximately 0.10%18.31%2.34%
Approximately 0.25%52.71%1.40%
Approximately 0.50%86.14%6.07%
1%95.74%7.01%
2%97.29%30.84%
5%97.58%14.02%
10%99.32%36.92%

This sweep varies a separate training mixture, with equal numbers of poison and cover examples. ASR uses the 1,032-case page-test set. Over-triggering is measured across 214 requests with DOM evidence of the corresponding controls. These are case-level rates on a different neighboring-request set from the N/U ablation, so comparisons are made within each table.

At 1% poisoning, ASR reaches 95.74% with 7.01% diagnostic over-triggering. At 5%, ASR increases to 97.58%, while over-triggering rises to 14.02%; at 10%, the corresponding rates are 99.32% and 36.92%. These results support interpreting elevated over-triggering in the high-ASR regime as a cost of pushing attack effectiveness toward saturation. The 2%, 5% and 10% results also show that the relationship is nonmonotonic: high ASR does not inevitably require high over-triggering, and this sweep does not identify a universally optimal dose.

The approximately 20% failures highlighted by the reviewer are therefore discussed as a selectivity limitation of the reported high-ASR setting, within a broader attack-success–selectivity trade-off. The N ablation shows that neighboring supervision can substantially improve this balance. We will add both tables and report attack success, neighboring activation and benign utility together. These conclusions concern COOKIE, the tested N/U amounts and one training seed, rather than establishing a general benefit or harm from U across settings.

AuthorsOpenReview ↗

Response part 4

Filtering the actual contaminated training corpus (W3/Q3).

We agree that detection should be evaluated on the contaminated training corpus itself. We therefore applied five filtering methods to the actual 20,000-row COOKIE training mixture, comprising 1,000 injected poison rows, 1,000 cover rows and 18,000 clean-component rows. Semantic scoring uses the instruction, screenshot and annotated action with a local Qwen3-VL-4B judge. Source labels are excluded from detector inputs and used to measure removal rates.

The table summarizes operating points with nonpoison-source removal below 30%. This is a retrospective reporting cap; the methods are not calibrated to an identical false-positive rate.

Filtering methodRequired accessPoison-source removal (TPR)Nonpoison-source removalTotal removed
Perplexity filteringCorpus + pretrained language scorer1.1% (11/1,000)1.86% (354/19,000)1.83% (365/20,000)
AlpaGasus-style GUI scoringCorpus + local 4B judge33.4% (334/1,000)8.52% (1,618/19,000)9.76% (1,952/20,000)
VDC-style GUI scoringCorpus + local 4B judge44.4% (444/1,000)21.44% (4,074/19,000)22.59% (4,518/20,000)
Spectral Signatures, global GUI adaptationCorpus + trained grounder activations31.1% (311/1,000)3.63% (689/19,000)5.00% (1,000/20,000)
Activation Clustering, global GUI adaptation, k=4Corpus + trained grounder activations0.0% (0/1,000)9.67% (1,838/19,000)9.19% (1,838/20,000)

The quality scorer retains 1,296 unassessed rows in the corpus; all rates use the full denominators.

These results show that the injected rows contain detectable signals, while the operating points reported under this removal cap achieve partial rather than complete removal. VDC-style scoring detects the largest fraction, 44.4%, leaving 556 injected rows while removing 4,074 nonpoison rows. Spectral screening removes 31.1% of injected rows at 3.63% nonpoison-source removal, increasing the poison fraction among removed rows from 5% to 31.1%.

The first three methods can screen data before training the suspect grounder; Spectral and AC require its learned activations and serve as offline corpus analyses. These metrics measure injected-source identification: nonpoison-source removal does not necessarily mean removal of a verified-correct annotation. We will add this corpus-level evaluation to Detection Evaluation and revise the blanket detection-resistance interpretation to reflect these detection–removal trade-offs. These measurements quantify sample-filtering performance; downstream ASR after training on filtered data is outside this evaluation.

Reviewer_MXMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a data-poisoning attack on GUI grounding models. Instead of using a fixed visual or text trigger, the attacker poisons training examples so that instructions with a certain intent are mapped to an attacker-chosen UI element. The evaluation covers three main components like cookie consent with three model backbones. The paper also tests existing backdoor detectors.

Critical strengths
  • The paper studies a meaningful attack surface. A wrong grounding decision can cause a GUI agent to take an action that conflicts with the user's request.

  • The host-disjoint page evaluation and results across three backbones provide useful evidence that poisoning can transfer beyond the training pages.

Critical weaknesses
  • The central claim of intent-level generalization is not yet well established. The held-out paraphrase evaluation uses only seven phrasings per scenario, written by the same authors who created the training phrases. The training data already contains hundreds of paraphrases. The good performance on these few held-out phrasings does not show that the backdoor works for a broad, independently sampled class of user intents. This is the main claimed difference from prior backdoors.

  • The detector evaluation does not convincingly test the most relevant defense. The most compared detectors look for visible text or image triggers. This attack deliberately uses neither, so their failure is not very surprising. The relevant signal is the mismatch between the user's intent and the selected UI element. The semantic audit uses small local judges and even treats a decline-tracking instruction paired with an ``Accept all'' button as plausible. This does not support a strong claim that the attack is hard to detect with appropriate semantic checks.

Questions for authors
  1. Can the authors test the attack on a substantially larger set of independently written user instructions, including less similar ways of expressing the same intent?

  2. How does the attack perform against a strong instruction-to-selected-element consistency check at inference time, with a reported false-positive rate on benign requests?

2 comments
AuthorsOpenReview ↗

Response part 1

Thank you for identifying the two central evidence gaps: coverage of independently expressed intents and evaluation against a relevant semantic defense. We expanded COOKIE testing and directly evaluated instruction–action consistency, as detailed below.

Independent expressions and semantic generalization (W1/Q1).

We agree that seven author-written phrasings provide limited evidence for expression generalization. We expanded COOKIE testing to 180 same-intent instructions independently generated by GLM, DeepSeek and Kimi, 60 each, without providing the training bank, original seven test phrasings or grounding predictions. Generation was followed by duplicate/overlap filtering and intent-consistency checks by a model other than the generator. These are model-generated expressions rather than a sample of human-user language.

On the same 740 instruction–page cases across 30 pages, instruction-averaged payload-hit rate is 96.6% for the poisoned model versus 9.3% for SFT, an increase of 87.3 percentage points (95% CI: 80.1–94.5). Thus, the new expressions activate the poisoned model well beyond its unpoisoned counterpart’s existing error tendency.

To examine less training-similar expressions, we measured mean cosine distance to the 504 unique expressions used in poison training with Qwen3-Embedding-0.6B:

Distance groupMean-distance rangeInstructionsPoisoned ASRSFT payload-hit rateIncrease, pp [95% CI]
G10.222–0.2643497.8%6.9%+90.9 [83.3, 97.9]
G20.266–0.2832397.4%12.6%+84.8 [74.2, 94.8]
G30.286–0.3204197.7%6.1%+91.5 [85.6, 97.9]
G40.322–0.3562998.4%13.3%+85.1 [73.5, 95.7]
G50.359–0.4072995.0%12.0%+83.0 [70.8, 94.5]
G60.411–0.5412492.3%7.2%+85.1 [77.4, 92.7]

Rates are averaged equally across instructions. Confidence intervals use paired page-cluster bootstrap. Distance stratification is exploratory. Increases are computed from unrounded rates.

The most distant mean-distance group retains 92.3% ASR, compared with 7.2% for SFT, an increase of 85.1 points [77.4, 92.7]. As a separate check against similarity to a single training phrase, we also grouped instructions by their distance to the nearest training expression. The farthest group contains 49 instructions, with nearest-expression distances of 0.206–0.400; poisoned ASR is 93.7%, versus 12.3% for SFT, an increase of 81.3 points [71.3, 91.7].

The source-wise results provide a complementary check of independence from a single generator:

Independent sourceInstructionsPoisoned ASRSFT payload-hit rateIncrease, pp [95% CI]
GLM6096.3%10.8%+85.6 [76.5, 94.8]
DeepSeek6096.9%7.7%+89.2 [83.2, 94.9]
Kimi6096.7%9.6%+87.1 [78.6, 95.5]
All sources18096.6%9.3%+87.3 [80.1, 94.5]

The same instruction averaging and paired page-bootstrap intervals apply; increases use unrounded rates.

These results support within-intent expression transfer on COOKIE across the tested independent sources. They also show that the high ASR persists under a nearest-expression similarity check, rather than relying only on average distance. The slight attenuation at the distant end is not strictly monotonic and does not define an activation cutoff. We will add the paired comparisons to Main Results and distinguish this evidence from generalization to different intents or representative human-user populations.

AuthorsOpenReview ↗

Response part 2

Inference-time instruction–action consistency (W2/Q2).

We agree that instruction–selected-element consistency is the relevant inference-time defense. Since our attack targets a compact 2B GUI grounder, we first evaluate local Qwen3-VL-4B and Qwen3-VL-8B checkers, then examine stronger API judges.

Each checker receives the user instruction, the screenshot with the predicted click marked, and a local crop, without the correct target or attack labels. We evaluate 300 requests (150 target-intent, 75 neighboring-intent and 75 other legitimate requests), checking both poisoned-model and SFT predictions: 600 actions per checker. The reference labels contain 213 noncompliant, 319 compliant and 68 unassessable actions; TPR/FPR use the first two groups. The common policy blocks inconsistent or uncertain judgments. Accepted ASR counts payload hits that pass the check among all 150 target-intent requests; the unchecked ASR on this subset is 96.0%. This is a separate evaluation from the 740-case independent-expression test.

Local visual checkerNoncompliant blocking, TPRCompliant false blocking, FPRAccepted payload-hit ASR
Qwen3-VL-4B58.7% (125/213)12.5% (40/319)96.0% → 37.3% (56/150)
Qwen3-VL-8B81.2% (173/213)11.3% (36/319)96.0% → 9.3% (14/150)

Local checking reduces attack success, but 56 and 14 target-intent requests still produce accepted payload hits with the 4B and 8B checkers. Their accepted SFT payload-hit rates on the same requests are 4.0% and 2.0%, respectively. The checkers also incorrectly block 12.5% and 11.3% of compliant actions. These results quantify residual risk and the benign-action cost under local checking.

We additionally evaluate three high-capability visual API judges under the same conditions:

API visual judgeBlocking TPRFalse-blocking FPRAccepted payload-hit ASRCheck latency p50 / p95Estimated USD / 1,000 checks
DeepSeek Flash94.4% (201/213)7.8% (25/319)1.3% (2/150)1.18 / 14.22 s$0.23
Kimi K393.0% (198/213)4.4% (14/319)2.7% (4/150)4.65 / 10.35 s$3.31
GLM-5.3-Flash93.4% (199/213)6.3% (20/319)4.0% (6/150)2.41 / 4.83 s$0.26

TPR/FPR reflect the blocking policy, including insufficient-evidence decisions.

API costs are estimated from measured token usage and listed prices per 1,000 checks. Price references dated November 10, 2026: DeepSeek, Kimi, GLM. Latencies measure judge calls.

The stronger judges reduce accepted ASR to 1.3–4.0%, with 4.4–7.8% false blocking. For comparison, median grounding latency is 0.806 s in a small benchmark of 54 observations; the table reports the additional checker-call latency. Blocking prevents a click without replanning, so lower accepted ASR measures blocked malicious actions rather than restored task completion.

The semantic-audit results in Appendix A.4 require correction. We will replace the affected results with the direct consistency evaluation above and revise the associated stealth interpretation: local checking leaves residual attacks in this setting, while stronger semantic checks are effective defenses with measured false blocking, latency and inference cost.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response [1/2]

We thank all three reviewers for their constructive feedback. We have provided point-by-point responses and additional experiments. At the time of writing, we have not yet received follow-up feedback. We would welcome further discussion with the reviewers and the Area Chair. Below, we summarize the main questions and the evidence addressing them.

  1. Q1. What is the novelty beyond existing semantic-backdoor methods? — fsXZ W1/Q1 and W4/Q4; Dm2m W3.
  2. Q2. How does the attack generalize to independent expressions and other GUI scenarios? — MXMS W1/Q1; fsXZ W1/Q1; Dm2m W2.
  3. Q3. Can a defense directly checking instruction–action consistency stop the attack? — MXMS W2/Q2; Dm2m W1.
  4. Q4. Can poisoned examples be detected inside the actual contaminated training corpus? — fsXZ W3/Q3.
  5. Q5. What controls over-triggering, and do N and U have the same effect? — fsXZ W2/Q2.

GUI grounding maps an instruction and screenshot to a click location. GistBackdoor poisons training coordinates so that a legitimate intent, such as declining tracking, selects a harmful native control, such as Accept All. Inference-time instructions and screenshots are unchanged. Attack success rate (ASR) measures redirection to that control.

Q1. What is the novelty beyond existing semantic-backdoor methods?

The novelty lies in establishing and empirically characterizing intent-triggered backdoor injection in GUI grounding: its feasibility, expression transfer, selectivity and defense boundaries. Three contributions support this claim:

  • An attack surface: an ordinary action intent activates a backdoor in the component deciding where an agent clicks, producing an action that directly conflicts with the user's goal.
  • A grounding-specific construction: the attacker pairs a harmful native control with a conflicting legitimate intent whose correct alternative exists on the page. Poisoned labels follow the payload's location on each screenshot.
  • Empirical validation of attack feasibility and detection limits: the experiments establish transfer to independent expressions and measure neighboring-intent activation, poisoned-sample removal and residual attacks under semantic checking.

The key finding is that a legitimate user goal can become a backdoor activation condition that persists across independently generated expressions. In COOKIE, the effect remains far above clean fine-tuning even for less training-similar wording. This creates a security risk for GUI agents processing ordinary instructions and screenshots, and the selectivity and defense experiments provide evidence for evaluating and mitigating that risk.

This work builds on semantic-conditioned predecessors, including VPI, whose classification we will correct. VPI can encompass action intents; its grounding adaptation can use known target boxes to supply manipulated coordinate labels directly. For attack comparisons, we will report effectiveness under each method's own trigger and activation under ordinary intent instructions.

Q2. How does the attack generalize to independent expressions and other GUI scenarios?

We find that independently generated expressions of the same intent still trigger the attack on COOKIE, and outline its construction in additional GUI scenarios. We tested 180 independent expressions, obtaining 96.6% ASR versus 9.3% for clean fine-tuning (SFT). The same-intent expressions were generated by GLM, DeepSeek and Kimi, 60 each, without access to training expressions, original test phrasings or grounding predictions. Across 740 cases on 30 pages, the instruction-averaged increase is 87.3 percentage points (95% CI: 80.1–94.5). Poisoned ASR is 96.3–96.9% across the three sources.

Grouping by mean embedding distance to the 504 unique training expressions, the most distant group retains 92.3% ASR versus 7.2% for SFT. A separate nearest-expression analysis yields 93.7% versus 12.3% in its farthest group. These results support within-intent transfer to less training-similar wording.

For scenario extension, we apply the same attacker-driven rule—select a harmful native control and a conflicting legitimate intent with a correct alternative—to restore → permanently delete, save draft → publish, deny location access → allow, and cancel renewal → keep subscription. These pairs illustrate scenario construction; the quantitative evaluation covers COOKIE.

AuthorsOpenReview ↗

General Response [2/2]

Q3. Can a defense directly checking instruction–action consistency stop the attack?

Our attack still succeeds under local small-model consistency checks, while stronger API-based checks substantially mitigate it. We evaluate defenses for a 2B grounder, testing local checkers and stronger visual API judges on the same inputs. On 300 requests, checking poisoned-model and SFT outputs gives 600 actions per checker. Judgments are based on the instruction, marked screenshot and local crop. Blocking and false-blocking rates are measured on 213 noncompliant and 319 compliant reference actions, respectively; 68 reference-unassessable actions are excluded from these rates.

Consistency checkerNoncompliant-action blockingCompliant-action false blockingAccepted ASR
Local Qwen3-VL-4B58.7%12.5%37.3%
Local Qwen3-VL-8B81.2%11.3%9.3%
DeepSeek Flash94.4%7.8%1.3%
Kimi K393.0%4.4%2.7%
GLM-5.3-Flash93.4%6.3%4.0%

The defense blocks actions judged inconsistent or uncertain. Accepted ASR measures payload hits that pass this check among all 150 target-intent requests. Local 4B and 8B checks leave 37.3% and 9.3% accepted ASR, respectively, compared with 4.0% and 2.0% for checked SFT outputs.

Stronger API checkers reduce accepted ASR from 96.0% to 1.3–4.0%, with 4.4–7.8% false blocking. They add median call latency of 1.18–4.65 seconds, compared with 0.806 seconds for the 2B grounder in a separate 54-observation benchmark. We will incorporate these results into Detection Evaluation and replace the earlier Appendix A.4 audit results.

Q4. Can poisoned examples be detected inside the actual contaminated training corpus?

We find that the evaluated corpus filters leave substantial numbers of injected examples after filtering at the reported operating points. We tested perplexity filtering, AlpaGasus-style and VDC-style scoring, Spectral Signatures, and Activation Clustering on the actual 20,000-example mixture: 1,000 injected and 19,000 nonpoison-source examples. The reported operating points were selected retrospectively using a nonpoison-source removal cap of 30%; each method's removal rates are reported separately.

VDC-style scoring achieves the highest injected-example removal rate among these operating points, 44.4%, leaving 556 of 1,000 injected examples (55.6%) while removing 21.44% of nonpoison-source examples. Spectral Signatures removes 31.1% and 3.63%, respectively, leaving 689 injected examples.

This experiment measures corpus-filtering performance using injection-source labels as the reference.

Q5. What controls over-triggering, and do N and U have the same effect?

Neighboring-intent supervision improves selectivity; the tested unrelated-task block provides no unrelated-task accuracy gain. A fixed-budget four-way study compares neither block, N only, U only, and N+U. N contains neighboring cookie-management requests; U contains unrelated site tasks. Without U, adding N improves neighboring-task accuracy from 77.03% to 91.89% and reduces over-triggering from 16.22% to 1.35%, while ASR changes from 96.41% to 91.86%. With U, N likewise improves accuracy from 74.32% to 90.54% and reduces over-triggering from 14.86% to 2.70%.

This single-seed supplementary study uses 500 common cover examples per arm, versus 1,000 in the original recipe, with clean-data replacement for omitted N/U examples. We will revise Method to distinguish the original cover construction from the explicitly separated N/U blocks in this study. The original model's approximately 20% activation on selected neighboring requests motivates targeted negative supervision at this high-ASR operating point. The separate dose analysis also quantifies a nonmonotonic attack-success–selectivity trade-off.

Together, the results establish that legitimate user goals can activate GUI grounding backdoors and show how targeted supervision and semantic checks change the resulting risk.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.