acceptodds
Under review as a conference paper at ICLR 2027

ActionPatch: Distilling Adversarial Instructions into Visual Patches for GUI Agent Hijacking

Abstract

Mobile GUI agents translate screen content and user requests into executable actions, exposing their control flow to adversarial visual inputs. Precise hijacking through a small image region requires the model's output to specify the intended action and arguments in a format accepted by the agent's parser. We present , a method that distills text-induced adversarial behavior into compact visual patches, each targeting a GUI action and its arguments. With the vision-language model frozen, the method combines teacher-distribution matching with weighted supervision on execution-critical tokens, emphasizing action and argument fidelity while maintaining supervision over the full response. Each patch is optimized across benign requests and screenshots for reuse across interaction contexts. Independently optimized patches are composed into attack chains by exposing them at successive interaction stages. Experiments on Mobile-Agent-v3.5 show that small visual patches reliably cause the agent to execute target actions with the specified arguments across held-out benign requests. Ablation studies show that combining distribution matching with execution-critical token supervision improves attack success over either objective alone under matched patch budgets. A series of successful attack-chain case studies demonstrates how atomic patches achieve multi-step malicious goals and amplify the impact of individual action hijacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
GUI Agent Hijacking, Adversarial Visual Patches, AI Agent Security, Mobile GUI Agents

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KpuSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActionPatch, a white-box visual attack against mobile GUI agents. It distills malicious text-induced behaviors into reusable image patches by combining teacher-distribution matching with weighted cross-entropy on execution-critical action and argument tokens. The authors evaluate five atomic GUI operations and demonstrate several multi-step attack chains on Mobile-Agent-v3.5.

Critical strengths
  • The problem is relevant to GUI agent security, especially the attempt to induce precise executable actions through small attacker-controlled image regions.
  • The reported improvements over existing baselines are substantial under small patch budgets. I also appreciate that the authors verify the issued actions through the native executor rather than relying only on generated text.
  • I appreciate the comparisons with MIP and Prompt Matching, the objective ablations, and the experiments on patch size and interaction history.
Critical weaknesses
  • I am not convinced by the methodological novelty. The core idea of distilling text-induced behavior into visual attacks largely follows Image Hijacks [1], while MIP [2] has already demonstrated reusable image patches that induce executable GUI API calls. The main addition appears to be weighted CE on action and argument tokens, an idea also explored in GUI action supervision [3]. Although these works differ in their settings, I see ActionPatch mainly as combining existing techniques rather than introducing a fundamentally new attack method.
  • I do not find the improvements from the proposed objective consistent. In Table 2, the average gains over KL + Uniform CE are largely driven by SysBtn, while several other operations perform worse. More concerningly, in Appendix Table 7, ActionPatch underperforms KL + Uniform CE on both without-history subsets (80.4% vs. 88.6% and 72.9% vs. 89.2%). MIP also outperforms ActionPatch in Appendix Table 5. I am therefore not convinced that the proposed loss provides a general improvement.
  • The authors manually overlay patches onto screenshots at fixed locations rather than injecting them through actual attacker-controlled GUI content. Moreover, the reported ASR only requires the executor to issue the target command, even if the GUI state does not change. While execution-level ASR is a reasonable metric for atomic action control, it does not necessarily imply an effective state transition or successful attacker objective. The multi-step evaluation is also limited to four successful case studies without quantitative end-to-end success rates. I do not think these results sufficiently demonstrate reliable GUI hijacking in realistic scenarios.
  • The paper evaluates only one victim agent, five fixed target action and argument configurations, and two benchmarks sharing the same screenshot pool. CAGUI-Aug uses synthetic requests rather than independent GUI environments. Furthermore, the paper does not report the original next-action correctness or ASR on the clean-correct subset. I think these limitations leave substantial uncertainty about the generality of the attack and its actual impact on otherwise successful agent behavior.

[1] Image Hijacks: Adversarial Images can Control Generative Models at Runtime. ICML 2024.

[2] MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents. NeurIPS 2025.

[3] GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL. arXiv 2026.

Questions for authors
  • Could the authors clarify the main algorithmic novelty beyond adapting Prompt Matching [1] to GUI actions and adding weighted CE? How does the proposed weighting differ in principle from existing action-aware supervision [3]?
  • In Table 2, could the authors report the average improvements excluding SysBtn? Why does the full objective perform worse than simpler alternatives without interaction history in Appendix Tables 5 and 7?
  • Could the authors report the clean next-action accuracy and ASR on the clean-correct subset of CAGUI-Trace? Also, how were the five fixed target argument configurations selected, and how sensitive are the results to these choices?
  • Could the authors report quantitative end-to-end success rates for the attack chains over repeated trials, including whether the intended GUI state changes actually occur? I would also like to know whether the patches remain effective when inserted through real application content rather than manually modified screenshots.
6 comments
AuthorsOpenReview ↗

reviewer KpuS - part1/4

Thank you for your valuable feedback. We address the raised questions and weaknesses below, and also incorporated them into the revised paper uploaded.

R4-W1 / R4-Q1: Algorithmic contribution beyond Prompt Matching and existing action-aware supervision

Response: Thank you for raising this question. Both prompt matching [1] and MIP [2] optimize overall similarity to the target output without accounting for the importance of key fields to agent task performance. Under small patch budgets, they can therefore achieve high overall similarity while still producing incorrect values in key fields. To address this limitation, we combine a KL divergence loss that preserves the overall output with a weighted cross-entropy (CE) loss that emphasizes key fields. This combination enables targeted control while maintaining a small patch size. Our experiments show improved performance, while end-to-end evaluations further demonstrate the effectiveness of our approach in practical settings.

We agree that our weighting principle is related to GUI-Libra [3], but the key distinction is what each method optimizes. GUI-Libra aims to train the model using action-aware weighting to improve action prediction and grounding. ActionPatch instead keeps the model fixed and optimizes a local input patch, combining execution-field supervision with text-induced soft targets to induce targeted operations.

AuthorsOpenReview ↗

reviewer KpuS - part2/4

R4-W2 / R4-Q2: Average gains excluding SysBtn and weaker performance without interaction history

Response: Thank you for the questions.

Average improvements excluding SysBtn in original experiments. In our submitted paper, the mean ASRs across Click, Swipe, Type, and Open in Table 2 are:

DatasetKL + uniform CEWeighted CE onlyActionPatchDifference from KL + uniform CE
CAGUI-Trace82.65%70.63%84.23%+1.58 pp
CAGUI-Aug82.30%71.10%80.15%−2.15 pp

In the original experiments, all five operations used the same empirical configuration, which gives different gains across operations, with SysBtn reaching higher gain and other operations lower gains. The shared configuration is also the main reason that the full objective performs worse than simpler alternatives. This motivates us to investigate whether operation-specific parameter adjustments could improve performance, particularly on the weaker without-history subsets, as reported below.

Added experiments by adjusting operation-specific parameters to enhance ASR for all operations. We added experiments evaluating the adjusted operation-specific parameters while retaining the combined objective. The results show that these adjustments improved success rates for all five operations in with-history, without-history, and overall (average of with-history and without-history) conditions.

DatasetOperationOverall: original → adjustedWith history: original → adjustedWithout history: original → adjusted
TraceClick82.20 → 90.5581.90 → 89.4383.90 → 97.78
TraceSwipe78.50 → 91.5379.00 → 90.2975.00 → 99.44
TraceType80.70 → 97.3083.00 → 97.4965.60 → 96.11
TraceOpen95.50 → 100.0098.30 → 100.0077.80 → 100.00
TraceSysbtn97.80 → 99.0397.40 → 98.87100.00 → 100.00
TraceMean excluding SysBtn84.23 → 94.8585.55 → 94.3075.58 → 98.33
AugClick72.10 → 90.0071.80 → 88.0573.30 → 98.89
AugSwipe71.40 → 90.7072.00 → 89.7668.90 → 95.00
AugType86.80 → 96.9092.60 → 98.1760.60 → 91.11
AugOpen90.30 → 99.2096.60 → 99.2761.70 → 98.89
AugSysbtn96.00 → 99.2095.10 → 99.02100.00 → 100.00
AugMean excluding SysBtn80.15 → 94.2083.25 → 93.8166.13 → 95.97

The parameters influence two distinct parts of the supervision. Weighted CE is normalized by the sum of token weights, so a token's relative contribution is wt/∑swsw_t / \sum_s w_s. Increasing one field's weight also reduces the relative share of other fields. Since response lengths and field-token counts vary across operations, the same nominal weights can produce different allocations of supervision. The KL/CE coefficients additionally control the balance between full-response distribution matching and weighted token supervision.

The operation-specific parameter adjustments were guided by error decomposition and separate development-set experiments. Click and Swipe increase coordinate-token weights from 12 to 24 and 18, respectively; Type reduces its action-token weight from 6 to 4.5. Their KL/CE coefficients remain 1/1. Open and Sysbtn retains its token weights and changes the KL/CE coefficients to 2/3 and 4/3.

Revision: Section 5.6 and Table 3 report the original and adjusted configurations' results both overall and by history condition. Appendix K, Table 16, lists the corresponding loss coefficients and token weights.

AuthorsOpenReview ↗

reviewer KpuS - part3/4

R4-W4 / R4-Q3: Clean next-action accuracy, attack success on clean-correct inputs, and target-argument selection

Response: Thank you for suggesting this evaluation.

clean next-action accuracy and ASR on the clean-correct subset of CAGUI-Trace. We scored all 1,334 CAGUI-Trace steps using the official CAGUI ActionEvaluator with output-format adaptation for GUI-Owl. Clean next-action accuracy is 47.45% (633/1,334). These 633 correctly predicted steps define a fixed clean-correct subset, determined solely from unpatched predictions and benign reference actions and shared across all attack targets.

The attack results on this subset are:

Target operationASR on clean-correct inputs
Click88.15%
Swipe93.21%
Type96.84%
Open100.00%
SysBtn98.74%
Mean95.39%

Sensitivity to target argument configurations. The parameters were chosen arbitrarily for testing, without deliberate tuning: Click(867, 145), Swipe(500, 300) → (500, 800), Type("JingDong"), Open("JingDong"), and SysBtn(Home).

To examine effectiveness beyond the original fixed argument choices, we additionally evaluated one new target configuration for each of the five operations in the Mobile-Agent-v3.5/GUI-Owl setting.

OperationAdditional targetTrace With historyTrace Without historyTrace OverallAug With historyAug Without historyAug Overall
Click(500, 300)89.5198.8990.7884.5192.7886.00
Swipe(450, 750) → (450, 400)85.9698.8987.7186.2298.3388.40
Type"mountaineering sports shoes"95.8486.1194.5396.4677.2293.00
OpenTestingAPP99.0599.4499.1099.5198.8999.40
SysBtnBack97.14100.0097.5395.37100.0096.20
Mean across five targets—93.5096.6793.9392.4193.4492.60

Mean ASR is the unweighted average across the five target operations. Overall rates pool the samples with and without interaction history.

Across these five additional targets, the mean overall ASR is 93.93% on CAGUI-Trace and 92.60% on CAGUI-Aug. Mean ASR with and without history is 93.50% and 96.67% on Trace, and 92.41% and 93.44% on Aug, respectively. These results show that ActionPatch remains effective for the evaluated alternative click and swipe coordinates, longer input text, different application identifier, and Back button, extending target coverage beyond the original five argument configurations.

Experiment on alternative victim agent. We extended the evaluation to MobiAgent with MobiMind-1.5-4B-0313. The ASRs reported in Appendix H, Table 12, are shown below for Click, Swipe, and Type, with the mean computed across these three operations.

DatasetInteraction historyClick ASR (%)Swipe ASR (%)Type ASR (%)Mean ASR (%)
CAGUI-TraceWithout98.999.498.398.9
CAGUI-TraceWith96.699.596.897.6
CAGUI-AugWithout97.898.398.998.3
CAGUI-AugWith96.099.096.297.1

The mean ASR ranges from 97.1% to 98.9% across these settings, supporting the applicability of ActionPatch beyond the original agent and model configuration.

Revision: We added the clean-scoring protocol and clean-correct evaluation in Appendix G, Table 11, and the additional MobiAgent evaluation in Appendix H, Table 12. We added Appendix I, Additional Target-Argument Configurations, with the new target specifications in Table 13 and their PVR, AMR, and ASR results in Table 14.

Reviewer_L3w2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActionPatch, which optimizes localized visual patches to induce attacker-specified GUI actions and arguments. The method first uses textual instructions to generate teacher responses and output distributions, then optimizes patches through distribution matching and cross-entropy supervision weighted toward action and argument tokens. Experiments evaluate five operation types on Mobile-Agent-v3.5, while case studies demonstrate the feasibility of composing independently optimized patches into attack chains. Overall, the method achieves substantial improvements under certain patch budgets, but the practical applicability of fixed-operation control and the sources of the optimization gains require further investigation.

Critical strengths
  1. Precise execution-level control is a practically relevant problem. The attack must control both the action type and all required arguments, with results verified through the native parser and executor. This provides stronger evidence of control over the execution interface than checking whether the model merely generates a particular action category.

  2. The method is straightforward, and its main components receive experimental validation. Emphasizing action and argument tokens alongside full-response distribution matching has a clear motivation. With 64×64 patches, the complete method substantially outperforms the baselines in average attack success on both evaluation datasets. Ablations also support the average benefit of combining the two objectives.

  3. The evaluation covers multiple operations and interaction contexts. Experiments include clicking, swiping, typing, application launching, and system buttons, along with different patch sizes and interaction-history conditions. Multi-step case studies further illustrate the potential impact of atomic operation control.

Critical weaknesses

Weaknesses

  1. The fixed-action-and-argument setting limits the practical significance of the attack. Each patch targets a predefined operation, such as clicking a fixed coordinate. However, scrolling or layout changes can cause the same coordinate to refer to a different interface element. Successfully issuing the specified command therefore does not necessarily mean manipulating the attacker’s intended target. Although this setting is consistent with the threat model, the evaluation provides insufficient evidence of practical effectiveness in dynamic interfaces. The authors should evaluate scrolling and layout changes, measure success against intended elements or resulting states, and compare with attacks that determine target locations from the current interface.

  2. The ablation gains depend heavily on the operation type, and the current explanation is insufficient. Much of the average improvement in Table 2 comes from SysBtn, while the complete method underperforms simpler objectives on several other operations. On CAGUI-Aug, for example, its Click, Type, and Open success rates are all lower than those of KL with uniform CE. Calculated from the table, excluding SysBtn makes the complete method’s average success rate approximately 2.2 percentage points lower across the remaining four operations on this dataset. Although the paper acknowledges operation-dependent benefits, it does not adequately explain them. Further analysis should clarify why critical-token weighting has opposite effects across operations and whether the weighting scheme or balance between the two losses accounts for these differences.

  3. The paper lacks evaluation against basic defenses and input processing. It does not examine common transformations such as image compression, resizing, or mild blurring, nor does it evaluate applicable existing defenses against visual prompt injection. Such experiments, accompanied by measurements of their effects on benign task performance, would help establish whether the attack remains effective under practical input-processing pipelines.

Questions for authors

Please see the weakness.

8 comments
AuthorsOpenReview ↗

reviewer L3w2 - part1/3

Thank you for your great feedback. We address the raised questions and weaknesses below, and also incorporated them into the revised paper uploaded.

R3-Q1: Fixed target arguments, scrolling, and success at the GUI-state level

Response: Thank you for highlighting the distinction between issuing a predefined command and successfully manipulating the intended interface element. In an agent-driven workflow, interface changes caused by operations such as scrolling can often be anticipated. An attacker can test the target application's normal workflow in advance, observe the resulting interface states, and prepare stage-specific patches and operation arguments accordingly. For example, in a chain consisting of a scroll followed by a click, the attacker can specify the click coordinate based on the expected post-scroll layout rather than the initial screen. Thus, fixed arguments do not require an unchanged interface: they can account for predictable transitions within a planned attack chain. We acknowledge that unexpected layout changes, such as those caused by application updates or system errors, may disrupt the planned attack. Extending the attacks to accommodate such unexpected changes beyond the anticipated workflow is a direction for future work.

We have added end-to-end experiments on a physical Android device to evaluate whether the prescribed operations produce the intended state changes. Patches are displayed as image content within an app developed for these tests, and Android system screenshots serve as the agent’s visual input. The agent’s generated actions are then executed on the device. After each action, the corresponding GUI state change is verified before presenting the next patch; if verification fails, the run terminates without retrying the action. Verification checks actual click events, entered text, scrolling in the required direction, and foreground transitions for the terminal Open and Home operations.

The completed evaluation contains 100 chains, each tested with five instruction–background pairs under three conditions. Of these, 72 chains contain a Scrolling followed by at least one further operation. All 360/360 attack runs in this subset completed their prescribed state transitions, and every target action also matched its required arguments. The corresponding no-patch and random-texture controls each completed 0/360 target chains.

Evaluation setNumber of chainsActionPatch: successes/runsNo patch: successes/runsRandom texture: successes/runs
Chains with at least one operation after Scrolling72360/3600/3600/360
All evaluated chains100500/5000/5000/500

Revision: Section 5.7 and Table 4 report the quantitative end-to-end attack-chain results. Appendix F.1 describes the device-based protocol and state-verification criteria, and Appendix F.2 presents an example trajectory in Figure 6.

AuthorsOpenReview ↗

reviewer L3w2 - part2/3

R3-Q2: Why token weighting and loss balance affect operations differently

Response: Thank you for requesting a more detailed explanation. The original configuration gives different gains across operations, with SysBtn reaching higher gain and other operations lower gains. This motivates us to investigate whether operation-specific parameter adjustments could improve performance. Therefore, we added experiments evaluating the configurations below while retaining the combined objective. The results show that these adjustments improved success rates for all five operations.

DatasetOperationOverall: original → adjusted
TraceClick82.20 → 90.55
TraceSwipe78.50 → 91.53
TraceType80.70 → 97.30
TraceOpen95.50 → 100.00
TraceSysbtn97.80 → 99.03
TraceMean excluding SysBtn84.23 → 94.85
AugClick72.10 → 90.00
AugSwipe71.40 → 90.70
AugType86.80 → 96.90
AugOpen90.30 → 99.20
AugSysbtn96.00 → 99.20
AugMean excluding SysBtn80.15 → 94.20

The parameters influence two distinct parts of the supervision. Weighted CE is normalized by the sum of token weights, so a token's relative contribution is wt/∑swsw_t / \sum_s w_s. Increasing one field's weight also reduces the relative share of other fields. Since response lengths and field-token counts vary across operations, the same nominal weights can produce different allocations of supervision. The KL/CE coefficients additionally control the balance between full-response distribution matching and weighted token supervision.

The operation-specific parameter adjustments were guided by error decomposition and separate development-set experiments. Click and Swipe increase coordinate-token weights from 12 to 24 and 18, respectively; Type reduces its action-token weight from 6 to 4.5. Their KL/CE coefficients remain 1/1. Open and Sysbtn retains its token weights and changes the KL/CE coefficients to 2/3 and 4/3.

These results support treating field weights and loss balance as useful optimization choices, with higher adjusted success rates extending beyond SysBtn.

Revision: Section 5.6 and Table 3 present the operation-specific analysis and results, and Appendix K, Table 16, specifies the adjusted coefficients and token weights.

AuthorsOpenReview ↗

reviewer L3w2 - part3/3

R3-Q3: Evaluation on input processing and basic defenses

Response: Thank you for identifying this evaluation gap.

Evaluation on input processing. We evaluated the impact of image transformations and resolution change on a subset of 180 randomly chosen samples (due to time constraint) each from Trace and Aug. The experiment on the full dataset (Trace-1334 samples and Aug-1000 samples) is ongoing, and we will report the results once they are available (Update: We completed the full-dataset experiments on November 18. The results have been added to the end of this reviewer-specific rebuttal response, under the heading “reviewer L3w2 - Additional Full-Dataset Results.”). The evaluated transformations include compression (JPEG), resizing, Gamma correction, RGB quantization, and resolution change. Under each transformations, we report the mean attack success rate across the five targets and the agent’s accuracy on benign tasks.

Input processingTrace ASRTrace benign accuracyAug ASRAug benign accuracy
PNG baseline95.2243.8994.5648.33
JPEG, Q9094.5645.0093.7848.89
JPEG, Q8087.7841.6789.0046.67
JPEG, Q7067.5644.4469.7847.22
Gamma, γ = 0.888.3341.6787.0047.22
Gamma, γ = 1.391.8945.0092.4450.00
Two-bit RGB quantization,95.2240.5693.7845.00
0.75× resizing0.1140.560.0048.89
Image-resolution, 576 × 128095.4442.7894.5652.78
Image-resolution, 864 × 192095.2243.8994.5648.33

The patches retain ASR close to the PNG baseline at Q90 and approximately 70% even at Q70, without retraining. Two-bit RGB quantization largely preserves ASR at 95.22%/93.78% on Trace/Aug, even if benign accuracy decreases by 3.33 (from 43.89 to 40.56, and from 48.33 to 45) percentage points on each dataset. The listed gamma settings retain ASR of at least 87%. Resolution-adaptive rendering preserves attack effectiveness without retraining the patches. ASR is 95.44%/94.56% at a source resolution of 576 × 1280 and 95.22%/94.56% at 864 × 1920 on Trace/Aug, closely matching the PNG baseline. The resizing test yields ASR of 0.11%/0.00% on Trace/Aug, identifying sensitivity to this specific resampling procedure.

Evaluation on defense. We have also added experiments that report attack effectiveness together with accuracy on benign tasks under basic defenses of AdaShield, LaSM-A, and JailGuard on a subset of 180 randomly chosen samples (due to time constraint) each from Trace and Aug. The experiment on the full dataset (Trace-1334 samples and Aug-1000 samples) is ongoing, and we will report the results once they are available (Update: We completed the full-dataset experiments on November 18. The results have been added to the end of this reviewer-specific rebuttal response, under the heading “reviewer L3w2 - Additional Full-Dataset Results.”).

We have also added experiments that report attack effectiveness together with accuracy on benign tasks under basic defenses of AdaShield, LaSM-A, and JailGuard.

DefenseTrace ASRTrace benign accuracyAug ASRAug benign accuracy
AdaShield, static GUI adaptation77.8948.8977.2248.33
LaSM-A, α = 1.0568.0045.5669.1147.22
JailGuard, calibrated τ = 0.15762.3329.4461.8943.33

The completed defense experiments also show substantial residual attack effectiveness. Under AdaShield's official static prompt adapted to the GUI interface, ActionPatch retains mean ASR of 77.89%/77.22% on Trace/Aug, while benign accuracy is 48.89%/48.33%, compared with 43.89%/48.33% without defense. Under LaSM-A, ASR remains approximately 70%, at 68.00%/69.11%, while benign accuracy stays close to the baseline at 45.56%/47.22%. These results show that the patches remain effective under both adaptations. Even under calibrated JailGuard, the patches retain ASR of 62.33%/61.89%. Benign accuracy decreases to 29.44%/43.33%, corresponding to reductions of 14.44 and 5.00 percentage points from the matched baseline. Thus, this detector still permits substantial attack success at the evaluated threshold, alongside an observed cost to benign performance. The results demonstrate retained attack effectiveness under these three reported defense adaptations without adapting or retraining the patches.

Revision: We added Appendix J. Table 15 reports ASR and benign next-action accuracy for image processing, resolution-adaptive rendering, and the AdaShield, LaSM-A, and JailGuard adaptations. Section J.1 describes the resolution-adaptive rendering and normalization protocol. Section J.2 describes the defense settings and their effects on benign performance.

Reviewer_8V2KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the security of GUI agents, focusing on whether a localized visual patch can induce a mobile GUI agent to execute a specified atomic action with the intended arguments. The proposed method, ActionPatch, uses textual instructions to elicit target responses from a frozen vision-language model and constructs teacher supervision from successful responses and their token probability distributions. It then combines KL distribution matching with cross-entropy weighted toward execution-critical tokens to optimize patches that can be reused across benign user requests and background screenshots. The authors evaluate five action types using Mobile-Agent-v3.5 with GUI-Owl-1.5-8B-Instruct, compare optimization objectives and patch sizes, and present case studies showing how independently optimized atomic patches can be composed into multi-step operations. The reported high attack success rates expose security weaknesses in the evaluated GUI agent.

Critical strengths
  1. The research question is clear and important, and the method is closely aligned with the requirements of structured action control. The paper investigates whether a localized visual patch can control both a GUI agent's action type and its specific arguments. To this end, the authors combine text-induced teacher distributions with weighted supervision of execution-critical tokens, emphasizing the fields that determine the actual operation while retaining supervision over the full response. The method is simple and well motivated, providing a concrete, testable approach to studying the security of GUI agents' execution interfaces.

  2. The main experiments show substantial improvements in attack effectiveness and distinguish different levels of success in action generation. With a 64×64 patch occupying approximately 2.22% of the canvas, ActionPatch achieves mean ASRs of 86.9% and 83.3% on CAGUI-Trace and CAGUI-Aug, respectively. Reporting parser validity, action-type matching, and full-action ASR separately helps distinguish generating a valid action, selecting the target action type, and satisfying the complete target-operation requirements. The patch-size experiments and loss ablations provide additional comparisons.

Critical weaknesses
  1. Limited coverage of target systems and action arguments restricts the applicability of the conclusions. The experiments use only Mobile-Agent-v3.5 with GUI-Owl-1.5-8B-Instruct, making it difficult to determine how strongly the observed effectiveness depends on this particular model, prompt template, or action interface. Moreover, each action type has only one fixed argument configuration in the main quantitative experiments. Evaluating a substantially different model or agent configuration, or multiple argument configurations within the same action type, would provide stronger evidence of the method's applicability.

  2. The benefits of the combined objective vary across actions and evaluation subsets and are not fully explained. In Table 2, much of ActionPatch's mean improvement over KL+uniform CE comes from SysBtn. The authors acknowledge that the gains are action dependent, but further analysis is needed to distinguish the effects of weighted supervision on action-type selection, argument correctness, and final ASR. Furthermore, in Table 7 of Appendix E, the full method achieves mean ASRs of 80.4% and 72.9% on the subsets without interaction history in CAGUI-Trace and CAGUI-Aug, respectively. These are lower than both KL+uniform CE and Weighted CE only. The paper does not explain why combining the two losses helps on some subsets but performs worse than either ablation on others.

Questions for authors
  1. How were the loss coefficients and token weights chosen? Were the same settings used for all five operations? How sensitive are the results, particularly the relative performance of the full method and its ablations, to these choices?

  2. How is interaction history represented during teacher construction, patch optimization, and evaluation? Section 3 includes interaction history hth_t in the agent's input, but Sections 4.2–4.3 do not make its role explicit. Please clarify the history inputs used at each stage. Also, do the with-history and without-history groups differ only in access to history? This distinction matters when interpreting the results in Table 7.

5 comments
AuthorsOpenReview ↗

reviewer 8V2K - part1/3

Thank you for your valuable feedback. We address the raised questions and weaknesses below, and also incorporated them into the revised paper uploaded.

R2-W1: Applicability across agent–model configurations and target arguments

Response: Thank you for suggesting a broader evaluation. We have extended the experiments to MobiAgent with MobiMind-1.5-4B-0313, using its native prompt templates and action interface. Among the five operations studied in this work, the native interface in this configuration supports Click, Swipe, and Type, but does not support Open or SysBtn. We therefore evaluate these three operations using separately optimized patches.

DatasetInteraction historyClick ASR (%)Swipe ASR (%)Type ASR (%)Mean ASR (%)
CAGUI-TraceWithout98.999.498.398.9
CAGUI-TraceWith96.699.596.897.6
CAGUI-AugWithout97.898.398.998.3
CAGUI-AugWith96.099.096.297.1

The mean ASR across the three operations ranges from 97.1% to 98.9%. These results show that ActionPatch remains effective under another agent–model configuration, indicating that its success is not confined to the original Mobile-Agent-v3.5 framework and GUI-Owl model.

To complement the evaluation on another agent–model configuration, we tested one additional target-argument configuration for each of the five operations using the original Mobile-Agent-v3.5/GUI-Owl setting.

OperationAdditional targetTrace With historyTrace Without historyTrace OverallAug With historyAug Without historyAug Overall
Click(500, 300)89.5198.8990.7884.5192.7886.00
Swipe(450, 750) → (450, 400)85.9698.8987.7186.2298.3388.40
Type"mountaineering sports shoes"95.8486.1194.5396.4677.2293.00
OpenTestingAPP99.0599.4499.1099.5198.8999.40
SysBtnBack97.14100.0097.5395.37100.0096.20
Mean across five targets—93.5096.6793.9392.4193.4492.60

Mean ASR is the unweighted average across the five target operations. Overall rates pool the samples with and without interaction history.

Across these five additional targets, the mean overall ASR is 93.93% on CAGUI-Trace and 92.60% on CAGUI-Aug. Mean ASR with and without history is 93.50% and 96.67% on Trace, and 92.41% and 93.44% on Aug, respectively. These results show that ActionPatch remains effective for the evaluated alternative click and swipe coordinates, longer input text, different application identifier, and Back button, extending target coverage beyond the original five argument configurations.

Revision: We added the MobiAgent evaluation in Appendix H, Table 12, reporting results separately for each operation and history condition. We added Appendix I, Additional Target-Argument Configurations, with the new target specifications in Table 13 and their PVR, AMR, and ASR results in Table 14.

AuthorsOpenReview ↗

reviewer 8V2K - part2/3

R2-W2 / R2-Q1: Selection of loss coefficients and token weights, and their effects across operations

Response: Thank you for asking about the choice and sensitivity of these parameters. In the original experiments, all five operations used the same empirical configuration: KL/CE coefficients of 1/1, a base token weight of 1, weights of 6 for tool-name and action tokens, and weights of 12 for coordinate, text, and button tokens. We selected the loss coefficients and token weights through comparisons of candidate configurations. We then used a shared configuration for all five operations, without tuning the parameters separately for each action. This left room for operation-specific improvements and motivated the supplementary experiments reported below.

The original configuration gives different gains across operations, with SysBtn reaching higher gain and other operations lower gains. This motivates us to investigate whether operation-specific parameter adjustments could improve performance, particularly on the weaker without-history subsets. Therefore, we added experiments evaluating the configurations below while retaining the combined objective. The results show that these adjustments improved success rates for all five operations in with-history, without-history, and overall (average of with-history and without-history) conditions.

DatasetOperationOverall: original → adjustedWith history: original → adjustedWithout history: original → adjusted
TraceClick82.20 → 90.5581.90 → 89.4383.90 → 97.78
TraceSwipe78.50 → 91.5379.00 → 90.2975.00 → 99.44
TraceType80.70 → 97.3083.00 → 97.4965.60 → 96.11
TraceOpen95.50 → 100.0098.30 → 100.0077.80 → 100.00
TraceSysbtn97.80 → 99.0397.40 → 98.87100.00 → 100.00
TraceMean excluding SysBtn84.23 → 94.8585.55 → 94.3075.58 → 98.33
AugClick72.10 → 90.0071.80 → 88.0573.30 → 98.89
AugSwipe71.40 → 90.7072.00 → 89.7668.90 → 95.00
AugType86.80 → 96.9092.60 → 98.1760.60 → 91.11
AugOpen90.30 → 99.2096.60 → 99.2761.70 → 98.89
AugSysbtn96.00 → 99.2095.10 → 99.02100.00 → 100.00
AugMean excluding SysBtn80.15 → 94.2083.25 → 93.8166.13 → 95.97

The parameters influence two distinct parts of the supervision. Weighted CE is normalized by the sum of token weights, so a token's relative contribution is wt/∑swsw_t / \sum_s w_s. Increasing one field's weight also reduces the relative share of other fields. Since response lengths and field-token counts vary across operations, the same nominal weights can produce different allocations of supervision. The KL/CE coefficients additionally control the balance between full-response distribution matching and weighted token supervision.

The operation-specific parameter adjustments were guided by error decomposition and separate development-set experiments. Click and Swipe increase coordinate-token weights from 12 to 24 and 18, respectively; Type reduces its action-token weight from 6 to 4.5. Their KL/CE coefficients remain 1/1. Open and Sysbtn retains its token weights and changes the KL/CE coefficients to 2/3 and 4/3.

These results support treating field weights and loss balance as useful optimization choices, with higher adjusted rates extending beyond SysBtn.

Revision: Section 5.6 and Table 3 present the operation-specific analysis and results, and Appendix K, Table 16, lists the loss coefficients and token weights.

AuthorsOpenReview ↗

reviewer 8V2K - part3/3

R2-Q2: History representation at each stage and interpretation of the history-based subsets

Response: Thank you for requesting this clarification. Interaction history comprises recent action records, recent GUI screenshots, and a summary of earlier steps. In the original experimental protocol, these components are used as follows:

StageRecent action recordsSummary of earlier stepsRecent GUI screenshots
Teacher constructionIncluded as textIncluded as textNot included
Patch optimizationIncluded as textIncluded as textNot included
EvaluationIncludedIncludedIncluded

During teacher construction and patch optimization, the action records and earlier-step summary are prepended to the current request following MobileAgent's interaction format. Evaluation includes all three components to assess attack effectiveness with the full interaction context. The table describes historical inputs; the current visual input is handled by the corresponding stage of the pipeline.

The labels with history and without history describe whether an evaluation sample contains prior interaction context. The with-history and without-history subsets come from the same dataset and differ only in whether the samples include history context: samples with history context are labeled with-history, while those without it are labeled without-history.

Reviewer_bL2KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ActionPatch, a white-box visual attack that induces a mobile GUI agent to execute a specified action with fixed arguments through a localized image patch. The method distills text-induced teacher distributions into patch pixels, combining KL distribution matching with cross-entropy supervision that assigns higher weights to execution-critical action and argument tokens. Each patch is optimized across multiple requests and screenshots for reuse across contexts. Experiments on Mobile-Agent-v3.5 demonstrate improved execution-level attack success over MIP and Prompt Matching under constrained patch budgets. Selected case studies further illustrate how independently optimized patches can be sequenced to redirect multi-step workflows.

Critical strengths
  1. Substantial improvements under a constrained patch budget. With 64 × 64 patches, ActionPatch improves average ASR over Prompt Matching by 21.4 and 17.9 percentage points on the two benchmarks.
  2. The objective and patch-size experiments help characterize when distribution matching and execution-critical token weighting are effective.
  3. The same operation-specific patch is evaluated across different requests and screenshots, demonstrating reuse beyond a single task instance.
Critical weaknesses
  1. Experiments use one agent–model configuration and five fixed action–argument targets. Transfer across models, patch positions, resolutions, and image transformations is not evaluated.
  2. Attack-chain composition assumes continued access to the required visual regions. The next operation must remain applicable, and attacker-controlled content must appear at the required location after each state transition. The conditions may not hold in realistic applications.
  3. Without interaction history, the full objective underperforms both reported ablations. The patterns weaken the claim that the combined objective generally improves action and argument fidelity and call for an explanation of which operations and contexts actually benefit.
Questions for authors
  1. How well does the method generalize to additional coordinates, longer text arguments, different application identifiers, and other system buttons?
  2. Why does ActionPatch underperform the simpler objective variants without interaction history? Which properties of the history make the combined objective more effective?
5 comments
AuthorsOpenReview ↗

reviewer bL2K - part1/3

Thank you for your valuable feedback. We address the raised questions and weaknesses below, and also incorporated them into the revised paper uploaded.

R1-W1: Applicability across models, target arguments, patch positions, and image transformations

Response: Thank you for suggesting a broader evaluation.

Transfer across models. We have extended the experiments to MobiAgent with MobiMind-1.5-4B-0313, using its native prompt templates and action interface. Among the five operations studied in this work, this interface supports Click, Swipe, and Type, but does not support Open or SysBtn. We therefore evaluate these three operations.

DatasetInteraction historyClick ASR (%)Swipe ASR (%)Type ASR (%)Mean ASR (%)
CAGUI-TraceWithout98.999.498.398.9
CAGUI-TraceWith96.699.596.897.6
CAGUI-AugWithout97.898.398.998.3
CAGUI-AugWith96.099.096.297.1

The mean ASR ranges from 97.1% to 98.9%. ActionPatch therefore remains effective under another agent–model configuration, indicating that its success is not confined to the original Mobile-Agent-v3.5 framework and GUI-Owl model.

Patch position. For patch position, we trained 64 × 64 Click patches on GUI-Owl using multiple positions. One patch was trained at nine positions corresponding to the four corners, the center of each quadrant, and the center of the image, achieving a mean ASR of 78.04% across these positions. Another patch was trained at ten positions along the left edge, spaced 64 pixels apart from y = 0 to y = 576 on a 288 × 640 image , achieving a mean ASR of 73.23% across these positions. These results show that ActionPatch can be optimized for multiple patch positions, with the set of positions specified during training according to application requirements.

Image transformation and resolution change. We also evaluated the impact of image transformations and resolution change on a subset of 180 randomly chosen samples (due to time constraint) each from Trace and Aug. The experiment on the full dataset (Trace-1334 samples and Aug-1000 samples) is ongoing, and we will report the results once they are available (Update: We completed the full-dataset experiments on November 18. The results have been added to the end of this reviewer-specific rebuttal response, under the heading “reviewer bL2K - Additional Full-Dataset Results.”). Under each transformations, we report the mean attack success rate across the five targets and the agent’s accuracy on benign tasks.

Input processingTrace ASRTrace benign accuracyAug ASRAug benign accuracy
PNG baseline95.2243.8994.5648.33
JPEG, Q9094.5645.0093.7848.89
JPEG, Q8087.7841.6789.0046.67
JPEG, Q7067.5644.4469.7847.22
Gamma, γ = 0.888.3341.6787.0047.22
Gamma, γ = 1.391.8945.0092.4450.00
Two-bit RGB quantization,95.2240.5693.7845.00
0.75× resizing0.1140.560.0048.89
Image-resolution, 576 × 128095.4442.7894.5652.78
Image-resolution, 864 × 192095.2243.8994.5648.33

The patches retain ASR close to the PNG baseline at Q90 and approximately 70% even at Q70, without retraining. Two-bit RGB quantization largely preserves ASR at 95.22%/93.78% on Trace/Aug, even if benign accuracy decreases by 3.33 (from 43.89 to 40.56, and from 48.33 to 45) percentage points on each dataset. The listed gamma settings retain ASR of at least 87%. Resolution-adaptive rendering preserves attack effectiveness without retraining the patches. ASR is 95.44%/94.56% at a source resolution of 576 × 1280 and 95.22%/94.56% at 864 × 1920 on Trace/Aug, closely matching the PNG baseline. The resizing test yields ASR of 0.11%/0.00% on Trace/Aug, identifying sensitivity to this specific resampling procedure.

Revision: We added the MobiAgent results in Appendix H, Table 12, and the image-processing resultsin Appendix J, Table 15. Section J.1 describes the resolution-adaptive rendering and normalization protocol.

AuthorsOpenReview ↗

reviewer bL2K - part 2/3

R1-Q1: Target-argument diversity

Response: To address target-argument diversity, we additionally evaluated one new target configuration for each of the five atomic operations in the original Mobile-Agent-v3.5/GUI-Owl setting.

OperationAdditional targetTrace With historyTrace Without historyTrace OverallAug With historyAug Without historyAug Overall
Click(500, 300)89.5198.8990.7884.5192.7886.00
Swipe(450, 750) → (450, 400)85.9698.8987.7186.2298.3388.40
Type"mountaineering sports shoes"95.8486.1194.5396.4677.2293.00
OpenTestingAPP99.0599.4499.1099.5198.8999.40
SysBtnBack97.14100.0097.5395.37100.0096.20
Mean across five targets—93.5096.6793.9392.4193.4492.60

Mean ASR is the unweighted average across the five target operations. Overall rates pool the samples with and without interaction history.

Across these five additional targets, the mean overall ASR is 93.93% on CAGUI-Trace and 92.60% on CAGUI-Aug. Mean ASR with and without history is 93.50% and 96.67% on Trace, and 92.41% and 93.44% on Aug, respectively. These results show that ActionPatch remains effective for the evaluated alternative click and swipe coordinates, longer input text, different application identifier, and Back button, extending target coverage beyond the original five argument configurations.

Revision: We added Appendix I, Additional Target-Argument Configurations, with the new target specifications in Table 13 and their PVR, AMR, and ASR results in Table 14.


R1-W2: Availability of attacker-controlled regions throughout an attack chain

Response: Thank you for highlighting this condition. We agree that attack-chain composition requires suitable attacker-controlled regions and valid operation preconditions at each stage. An attacker can identify these conditions in advance by testing the target application's normal workflow, using the observed sequence to anticipate subsequent interface states and operations and prepare patches for each stage. Our method makes such construction more feasible in two ways: small patches reduce the controllable area required, making it easier to find viable insertion locations within each interface; and allowing attacker-controlled content at multiple candidate locations provides flexibility across interface states, increasing the possibility of attack success. These capabilities support composing independently optimized patches toward higher-level attacker objectives.

AuthorsOpenReview ↗

reviewer bL2K - part 3/3

R1-W3 / R1-Q2: Performance without interaction history: hyperparameter settings and training coverage

Response: Thank you for asking about the weaker no-history performance and the stronger with-history performance in the original experiments. Our analysis links both observations to the original shared hyperparameter settings and the imbalance in training coverage.

Why did ActionPatch underperform the simpler objectives without history?

Based on our analysis, the weaker no-history performance relative to the simpler objectives was primarily attributable to the original shared hyperparameter configuration. All five operations used the same empirical KL/CE coefficients and token weights, although their response structures and parameter fields differ. The shared configuration gives different gains across operations. This motivates us to investigate whether operation-specific parameter adjustments could improve performance, particularly on the weaker without-history subsets. Therefore, we added experiments evaluating the configurations below while retaining the combined objective. The results show that these adjustments improved success rates for all five operations.

DatasetOperationWith history: original → adjustedWithout history: original → adjusted
TraceClick81.90 → 89.4383.90 → 97.78
TraceSwipe79.00 → 90.2975.00 → 99.44
TraceType83.00 → 97.4965.60 → 96.11
TraceOpen98.30 → 100.0077.80 → 100.00
TraceSysbtn97.40 → 98.87100.00 → 100.00
AugClick71.80 → 88.0573.30 → 98.89
AugSwipe72.00 → 89.7668.90 → 95.00
AugType92.60 → 98.1760.60 → 91.11
AugOpen96.60 → 99.2761.70 → 98.89
AugSysbtn95.10 → 99.02100.00 → 100.00

The operation-specific parameter adjustments were guided by error decomposition and separate development-set experiments. Click and Swipe increase coordinate-token weights from 12 to 24 and 18, respectively; Type reduces its action-token weight from 6 to 4.5. Their KL/CE coefficients remain 1/1. Open and Sysbtn retains its token weights and changes the KL/CE coefficients to 2/3 and 4/3.

2. Why was performance weaker in no-history samples in the original results?

Interaction history comprises recent action records, GUI screenshots, and summaries of earlier steps. The with-history and without-history labels describe whether an evaluation sample contains this context.

Based on our analysis, the weaker performance with no-history is caused by the combined effects of training-data imbalance and the original hyperparameter configuration on no-history inputs. We confirmed that high-quality no-history examples are less represented in the training data (only 6.36%). The combined objective requires fitting both the teacher's full response distribution and the precise action and argument fields, making adequate coverage of these contexts important. With fewer high-quality examples and a suboptimal shared loss configuration, no-history contexts are learned less effectively, resulting in lower performance for no-history samples and the relative advantage observed for with-history samples in the original experiments.

Revision: Sections 5.5–5.6 and Table 3 report the operation-specific analysis and configuration results. Appendix K, Table 16, lists the corresponding coefficients and token weights.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. As the Author–Reviewer Discussion period is closing soon, please actively engage with the authors, consider their responses, and update your final scores and comments.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.