- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GistBackdoor, which poisons GUI grounding fine-tuning data to associate naturally expressed user intents with attacker-chosen interface elements, without adding textual markers or modifying screenshots at inference time. The authors evaluate attack effectiveness, generalization across pages and phrasings, preservation of benign utility, and detection performance across three backbones and three real-webpage scenarios. Further discussion would be useful regarding the practical feasibility of injecting 5% poisoned data plus an additional 5% cover data in the main experiments, and whether user intent remains present in the instructions passed from the planner to the grounder.
Critical strengths
The paper investigates a valuable question: replacing explicit textual or visual triggers with naturally expressed user intent, without requiring additional cues at inference time. This offers an interesting perspective on supply-chain risks in GUI agents. The experiments include several careful design choices, including host-disjoint training and evaluation pages, SFT and SFT+cover controls to isolate the contribution of cover data, and reproductions of multiple baseline attacks under a shared configuration. Poisoning-rate sweeps, layer-block ablations, and linear probes provide additional analysis. Attack success rates increase substantially across all three backbones and three scenarios, while general grounding utility is largely preserved. The authors also discuss limitations transparently and provide a thorough ethics statement.
Critical weaknesses
-
The evidence for stealth remains insufficient. Most evaluated detectors target traces associated with explicit triggers, such as rare strings, syntactic templates, and image patches. Since GistBackdoor avoids these traces by design, the all-cross GistBackdoor row provides limited support for its stealth. The paper itself identifies semantic inconsistency between instruction intent and the selected element as a remaining signal, but leaves inference-time instruction–output consistency checking to future work. The claim that the attack is difficult to detect therefore lacks sufficient evaluation against a defense directly targeting this signal. The small judges in Appendix A.4 flag almost none of the poisoned samples while flagging many legitimate samples, with a false-positive rate reaching 66.5%. These anomalous results raise concerns about the reliability of the current semantic audit and do not establish that semantic auditing is ineffective. Evaluation with a stronger LLM/VLM judge would strengthen this conclusion.
-
The evaluation of generalization across phrasings remains limited. Each scenario uses only seven held-out paraphrases, written by the same authors who constructed the training expressions. Although the results demonstrate generalization to unseen phrasings, they do not sufficiently establish applicability to a broader range of natural expressions. Expanding the test set and incorporating expressions from independent authors or real users would provide stronger evidence of intent-trigger generalization.
-
The interpretation of attack comparisons needs a clearer scope. The ASRpage and ASRpara columns in Table 3(a) evaluate all attacks using GistBackdoor’s intent instructions, without supplying the triggers required by BadNets, VisualTrap, and CBA. Their low scores in these columns are therefore expected and do not indicate that these attacks are generally weaker. Although the “Own trigger” column reports performance under each attack’s activation conditions, the two ASR columns primarily show that fixed-trigger attacks do not generalize to these intent expressions; they do not establish the general superiority of GistBackdoor. Additional comparisons with semantic or topic-conditioned backdoors, beyond the existing syntactic HiddenKiller baseline, would help clarify the novelty boundary.
Questions for authors
Please see the weakness.
4 comments
Response part 2
Independent expression coverage (W2).
We agree that seven expressions written by the training-data authors provide limited evidence for broader expression coverage. We expanded COOKIE testing to 180 same-intent instructions from three independent model sources—GLM, DeepSeek and Kimi, 60 each. Generators received neither the training instruction bank, the original seven expressions nor grounding predictions. Duplicate/overlap filtering and intent-consistency checks by a model other than each expression's generator were applied. These are independently model-generated expressions, rather than independently collected human-user instructions.
The evaluation comprises 740 matched instruction–page cases across 30 pages. We report each source separately to show whether the effect depends on a particular generator:
| Independent source | Instructions | Poisoned ASR | SFT payload-hit rate | Increase, pp [95% CI] |
|---|---|---|---|---|
| GLM | 60 | 96.3% | 10.8% | +85.6 [76.5, 94.8] |
| DeepSeek | 60 | 96.9% | 7.7% | +89.2 [83.2, 94.9] |
| Kimi | 60 | 96.7% | 9.6% | +87.1 [78.6, 95.5] |
| All sources | 180 | 96.6% | 9.3% | +87.3 [80.1, 94.5] |
Rates are averaged equally over instructions; confidence intervals use paired page-cluster bootstrap. Each source contributes 60 expressions. Increases are computed from unrounded rates. These are comparisons on matched poisoned-model/SFT inputs.
The poisoned model reaches 96.3–96.9% ASR across all three sources, while SFT remains at 7.7–10.8%. Thus, the added evidence extends beyond a small author-written phrase set and beyond one generation source.
We also examine less training-similar wording using Qwen3-Embedding-0.6B and the 504 unique expressions used in poison training. The complete exploratory mean-distance grouping is:
| Distance group | Mean-distance range | Instructions | Poisoned ASR | SFT payload-hit rate | Increase, pp [95% CI] |
|---|---|---|---|---|---|
| G1 | 0.222–0.264 | 34 | 97.8% | 6.9% | +90.9 [83.3, 97.9] |
| G2 | 0.266–0.283 | 23 | 97.4% | 12.6% | +84.8 [74.2, 94.8] |
| G3 | 0.286–0.320 | 41 | 97.7% | 6.1% | +91.5 [85.6, 97.9] |
| G4 | 0.322–0.356 | 29 | 98.4% | 13.3% | +85.1 [73.5, 95.7] |
| G5 | 0.359–0.407 | 29 | 95.0% | 12.0% | +83.0 [70.8, 94.5] |
| G6 | 0.411–0.541 | 24 | 92.3% | 7.2% | +85.1 [77.4, 92.7] |
Rates are averaged equally across instructions; intervals use paired page-cluster bootstrap, and increases use unrounded rates. Distance stratification is exploratory.
The most distant group contains 24 expressions, with mean distances 0.411–0.541, and retains 92.3% ASR versus 7.2% for SFT—an increase of 85.1 points [95% CI: 77.4–92.7]. To additionally check similarity to a single training phrase, the farthest nearest-expression group contains 49 expressions, with nearest distances 0.206–0.400; ASR is 93.7% versus 12.3%, an increase of 81.3 points [71.3–91.7].
We will add the source-wise and distance-stratified results to Main Results. They support within-intent expression transfer on COOKIE, including less training-similar expressions, while representative human-user language and additional intent–payload pairs remain outside this new evaluation. Embedding distance is a similarity proxy, and the distant-group results define neither a universal activation cutoff nor a strictly monotonic distance–ASR relationship.
Response part 3
Comparison scope and task-adapted semantic reference (W3).
We agree that the low ASRpage/ASRpara values for explicit-trigger baselines do not establish general superiority. These columns evaluate unmodified intent instructions, where several baselines' required triggers are absent. Their low activation is therefore expected. The Own-trigger column instead evaluates each attack under its required activation conditions. We will separate these two questions in Table 3(a), its caption and the accompanying discussion.
For the task comparison, the available methods do not directly match the full intent-triggered, native-control setting. We therefore use task-adapted versions of representative attacks. We retain their characteristic trigger mechanisms and express their supervision in the grounding model’s coordinate format, using a common backbone and matched poisoning budget and fine-tuning settings.
| Method | Adaptation to grounding |
|---|---|
| BadNets | Append the textual trigger to a grounding instruction and supervise the attacker-selected control’s coordinates. |
| HiddenKiller | Rewrite grounding instructions into the trigger syntactic pattern and supervise the payload control’s coordinates. |
| VisualTrap | Apply the visual trigger to screenshots and retain the trigger patch’s location as the attack target. |
| CBA | Place the composite textual keys within a single grounding instruction, with coordinate supervision for the payload. |
VPI becomes comparable through the same task-adaptation process. We instantiate its semantic trigger scenario as requests to decline cookies and its desired response as selecting the page’s Accept control. In this verifiable setting, the known payload box already specifies the desired coordinate response, making teacher-generated replies unnecessary. VPI permits human annotators as well as a teacher model to provide manipulated responses (Yan et al., 2024, §3). Pairing the original instruction and screenshot with that coordinate implements its semantic-poisoning construction in grounding.
For the basic adaptation without additional task-specific benign cover, the resulting recipe coincides with our existing no-cover ablation: 19,000 general clean grounding examples plus 1,000 poisoned intent–coordinate pairs. We will identify this existing condition as a VPI-style grounding adaptation using known target coordinates, making the recipe correspondence explicit. This is a basic adaptation without extra scenario-specific cover; VPI itself is not defined by the absence of such cover. Comparing it with the cover-augmented recipe examines the effect of task-specific benign supervision within the shared semantic-poisoning framework.
This no-cover reference is distinct from the “neither N nor U” control in the N/U ablation, which retains the common benign cover.
These changes clarify the contribution boundary: the semantic-poisoning principle is shared with prior work, while our experiments characterize its use for intent-triggered GUI actions. The no-cover correspondence supplies a task-specific semantic reference; differences in activation conditions remain separate from effectiveness under each method's own trigger.
Official Comment by Reviewer
Thank you for the rebuttal. It has resolved my concern. I will maintain my score.
Response part 1
Thank you for the constructive assessment and for identifying the limits of the original stealth, expression-generalization and attack-comparison evidence. We address these points with direct semantic checks, independent expression testing and a clearer account of task adaptation and activation conditions.
Strong instruction–action consistency checking (W1).
We agree that the explicit-trigger detector matrix does not establish resistance to semantic auditing. The semantic-audit results reported in Appendix A.4 require correction and do not support the original broad stealth interpretation. We will replace the affected evidence with the direct consistency evaluation below.
To address the requested stronger defense, we evaluate DeepSeek Flash, Kimi K3 and GLM-5.3-Flash visual judges, directly checking whether the selected control satisfies the instruction. Each checker receives the user instruction, the screenshot with the predicted click marked, and a local crop, without the correct target or attack labels. We evaluate 300 requests (150 target-intent, 75 neighboring-intent and 75 other legitimate requests), checking both poisoned-model and SFT predictions: 600 actions per checker. The reference labels contain 213 noncompliant, 319 compliant and 68 unassessable actions; TPR/FPR use the first two groups. The common policy blocks inconsistent or uncertain judgments. Accepted ASR counts payload hits that pass the check among all 150 target-intent requests; the unchecked ASR on this subset is 96.0%. This is a separate evaluation from the 740-case independent-expression test.
TPR/FPR reflect the blocking policy, including insufficient-evidence decisions.
API costs are estimated from measured token usage and listed prices per 1,000 checks. Price references dated November 10, 2026: DeepSeek, Kimi, GLM. Latencies measure judge calls.
Strong, contextualized semantic checking is effective in this setting. It reduces accepted ASR from 96.0% to 1.3–4.0%, while falsely blocking 4.4–7.8% of compliant actions. The additional judge call has median latency 1.18–4.65 s, compared with 0.806 s for the 2B grounder in a small benchmark of 54 observations. These measurements describe effectiveness, false blocking and added inference cost. Blocking stops actions without replanning, so accepted ASR and successful task completion remain different outcomes.
As a complementary evaluation for compact deployments, we test local 4B and 8B visual checkers on the same actions:
Both local checkers mitigate the attack, with the 8B checker substantially stronger, but accepted ASR remains 37.3% and 9.3%, compared with 4.0% and 2.0% for checked SFT outputs. This local result complements the strong-model evidence; it does not support a blanket claim of resistance to semantic checking.
We will revise the stealth wording in the abstract and introduction, replace the affected Appendix A.4 evidence, and add direct consistency measurements to Detection Evaluation. Table 4 will describe the methods and criteria actually tested. The conclusion and Limitations will recognize contextualized semantic checking as an effective evaluated defense, with its measured false blocking and inference overhead.