- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ActionPatch, a white-box visual attack against mobile GUI agents. It distills malicious text-induced behaviors into reusable image patches by combining teacher-distribution matching with weighted cross-entropy on execution-critical action and argument tokens. The authors evaluate five atomic GUI operations and demonstrate several multi-step attack chains on Mobile-Agent-v3.5.
Critical strengths
- The problem is relevant to GUI agent security, especially the attempt to induce precise executable actions through small attacker-controlled image regions.
- The reported improvements over existing baselines are substantial under small patch budgets. I also appreciate that the authors verify the issued actions through the native executor rather than relying only on generated text.
- I appreciate the comparisons with MIP and Prompt Matching, the objective ablations, and the experiments on patch size and interaction history.
Critical weaknesses
- I am not convinced by the methodological novelty. The core idea of distilling text-induced behavior into visual attacks largely follows Image Hijacks [1], while MIP [2] has already demonstrated reusable image patches that induce executable GUI API calls. The main addition appears to be weighted CE on action and argument tokens, an idea also explored in GUI action supervision [3]. Although these works differ in their settings, I see ActionPatch mainly as combining existing techniques rather than introducing a fundamentally new attack method.
- I do not find the improvements from the proposed objective consistent. In Table 2, the average gains over
KL + Uniform CEare largely driven by SysBtn, while several other operations perform worse. More concerningly, in Appendix Table 7, ActionPatch underperformsKL + Uniform CEon both without-history subsets (80.4% vs. 88.6% and 72.9% vs. 89.2%). MIP also outperforms ActionPatch in Appendix Table 5. I am therefore not convinced that the proposed loss provides a general improvement. - The authors manually overlay patches onto screenshots at fixed locations rather than injecting them through actual attacker-controlled GUI content. Moreover, the reported ASR only requires the executor to issue the target command, even if the GUI state does not change. While execution-level ASR is a reasonable metric for atomic action control, it does not necessarily imply an effective state transition or successful attacker objective. The multi-step evaluation is also limited to four successful case studies without quantitative end-to-end success rates. I do not think these results sufficiently demonstrate reliable GUI hijacking in realistic scenarios.
- The paper evaluates only one victim agent, five fixed target action and argument configurations, and two benchmarks sharing the same screenshot pool. CAGUI-Aug uses synthetic requests rather than independent GUI environments. Furthermore, the paper does not report the original next-action correctness or ASR on the clean-correct subset. I think these limitations leave substantial uncertainty about the generality of the attack and its actual impact on otherwise successful agent behavior.
[1] Image Hijacks: Adversarial Images can Control Generative Models at Runtime. ICML 2024.
[2] MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents. NeurIPS 2025.
[3] GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL. arXiv 2026.
Questions for authors
- Could the authors clarify the main algorithmic novelty beyond adapting Prompt Matching [1] to GUI actions and adding weighted CE? How does the proposed weighting differ in principle from existing action-aware supervision [3]?
- In Table 2, could the authors report the average improvements excluding SysBtn? Why does the full objective perform worse than simpler alternatives without interaction history in Appendix Tables 5 and 7?
- Could the authors report the clean next-action accuracy and ASR on the clean-correct subset of CAGUI-Trace? Also, how were the five fixed target argument configurations selected, and how sensitive are the results to these choices?
- Could the authors report quantitative end-to-end success rates for the attack chains over repeated trials, including whether the intended GUI state changes actually occur? I would also like to know whether the patches remain effective when inserted through real application content rather than manually modified screenshots.
6 comments
reviewer KpuS - part2/4
R4-W2 / R4-Q2: Average gains excluding SysBtn and weaker performance without interaction history
Response: Thank you for the questions.
Average improvements excluding SysBtn in original experiments. In our submitted paper, the mean ASRs across Click, Swipe, Type, and Open in Table 2 are:
| Dataset | KL + uniform CE | Weighted CE only | ActionPatch | Difference from KL + uniform CE |
|---|---|---|---|---|
| CAGUI-Trace | 82.65% | 70.63% | 84.23% | +1.58 pp |
| CAGUI-Aug | 82.30% | 71.10% | 80.15% | −2.15 pp |
In the original experiments, all five operations used the same empirical configuration, which gives different gains across operations, with SysBtn reaching higher gain and other operations lower gains. The shared configuration is also the main reason that the full objective performs worse than simpler alternatives. This motivates us to investigate whether operation-specific parameter adjustments could improve performance, particularly on the weaker without-history subsets, as reported below.
Added experiments by adjusting operation-specific parameters to enhance ASR for all operations. We added experiments evaluating the adjusted operation-specific parameters while retaining the combined objective. The results show that these adjustments improved success rates for all five operations in with-history, without-history, and overall (average of with-history and without-history) conditions.
| Dataset | Operation | Overall: original → adjusted | With history: original → adjusted | Without history: original → adjusted |
|---|---|---|---|---|
| Trace | Click | 82.20 → 90.55 | 81.90 → 89.43 | 83.90 → 97.78 |
| Trace | Swipe | 78.50 → 91.53 | 79.00 → 90.29 | 75.00 → 99.44 |
| Trace | Type | 80.70 → 97.30 | 83.00 → 97.49 | 65.60 → 96.11 |
| Trace | Open | 95.50 → 100.00 | 98.30 → 100.00 | 77.80 → 100.00 |
| Trace | Sysbtn | 97.80 → 99.03 | 97.40 → 98.87 | 100.00 → 100.00 |
| Trace | Mean excluding SysBtn | 84.23 → 94.85 | 85.55 → 94.30 | 75.58 → 98.33 |
| Aug | Click | 72.10 → 90.00 | 71.80 → 88.05 | 73.30 → 98.89 |
| Aug | Swipe | 71.40 → 90.70 | 72.00 → 89.76 | 68.90 → 95.00 |
| Aug | Type | 86.80 → 96.90 | 92.60 → 98.17 | 60.60 → 91.11 |
| Aug | Open | 90.30 → 99.20 | 96.60 → 99.27 | 61.70 → 98.89 |
| Aug | Sysbtn | 96.00 → 99.20 | 95.10 → 99.02 | 100.00 → 100.00 |
| Aug | Mean excluding SysBtn | 80.15 → 94.20 | 83.25 → 93.81 | 66.13 → 95.97 |
The parameters influence two distinct parts of the supervision. Weighted CE is normalized by the sum of token weights, so a token's relative contribution is . Increasing one field's weight also reduces the relative share of other fields. Since response lengths and field-token counts vary across operations, the same nominal weights can produce different allocations of supervision. The KL/CE coefficients additionally control the balance between full-response distribution matching and weighted token supervision.
The operation-specific parameter adjustments were guided by error decomposition and separate development-set experiments. Click and Swipe increase coordinate-token weights from 12 to 24 and 18, respectively; Type reduces its action-token weight from 6 to 4.5. Their KL/CE coefficients remain 1/1. Open and Sysbtn retains its token weights and changes the KL/CE coefficients to 2/3 and 4/3.
Revision: Section 5.6 and Table 3 report the original and adjusted configurations' results both overall and by history condition. Appendix K, Table 16, lists the corresponding loss coefficients and token weights.
reviewer KpuS - part3/4
R4-W4 / R4-Q3: Clean next-action accuracy, attack success on clean-correct inputs, and target-argument selection
Response: Thank you for suggesting this evaluation.
clean next-action accuracy and ASR on the clean-correct subset of CAGUI-Trace. We scored all 1,334 CAGUI-Trace steps using the official CAGUI ActionEvaluator with output-format adaptation for GUI-Owl. Clean next-action accuracy is 47.45% (633/1,334). These 633 correctly predicted steps define a fixed clean-correct subset, determined solely from unpatched predictions and benign reference actions and shared across all attack targets.
The attack results on this subset are:
| Target operation | ASR on clean-correct inputs |
|---|---|
| Click | 88.15% |
| Swipe | 93.21% |
| Type | 96.84% |
| Open | 100.00% |
| SysBtn | 98.74% |
| Mean | 95.39% |
Sensitivity to target argument configurations. The parameters were chosen arbitrarily for testing, without deliberate tuning: Click(867, 145), Swipe(500, 300) → (500, 800), Type("JingDong"), Open("JingDong"), and SysBtn(Home).
To examine effectiveness beyond the original fixed argument choices, we additionally evaluated one new target configuration for each of the five operations in the Mobile-Agent-v3.5/GUI-Owl setting.
| Operation | Additional target | Trace With history | Trace Without history | Trace Overall | Aug With history | Aug Without history | Aug Overall |
|---|---|---|---|---|---|---|---|
| Click | (500, 300) | 89.51 | 98.89 | 90.78 | 84.51 | 92.78 | 86.00 |
| Swipe | (450, 750) → (450, 400) | 85.96 | 98.89 | 87.71 | 86.22 | 98.33 | 88.40 |
| Type | "mountaineering sports shoes" | 95.84 | 86.11 | 94.53 | 96.46 | 77.22 | 93.00 |
| Open | TestingAPP | 99.05 | 99.44 | 99.10 | 99.51 | 98.89 | 99.40 |
| SysBtn | Back | 97.14 | 100.00 | 97.53 | 95.37 | 100.00 | 96.20 |
| Mean across five targets | — | 93.50 | 96.67 | 93.93 | 92.41 | 93.44 | 92.60 |
Mean ASR is the unweighted average across the five target operations. Overall rates pool the samples with and without interaction history.
Across these five additional targets, the mean overall ASR is 93.93% on CAGUI-Trace and 92.60% on CAGUI-Aug. Mean ASR with and without history is 93.50% and 96.67% on Trace, and 92.41% and 93.44% on Aug, respectively. These results show that ActionPatch remains effective for the evaluated alternative click and swipe coordinates, longer input text, different application identifier, and Back button, extending target coverage beyond the original five argument configurations.
Experiment on alternative victim agent. We extended the evaluation to MobiAgent with MobiMind-1.5-4B-0313. The ASRs reported in Appendix H, Table 12, are shown below for Click, Swipe, and Type, with the mean computed across these three operations.
| Dataset | Interaction history | Click ASR (%) | Swipe ASR (%) | Type ASR (%) | Mean ASR (%) |
|---|---|---|---|---|---|
| CAGUI-Trace | Without | 98.9 | 99.4 | 98.3 | 98.9 |
| CAGUI-Trace | With | 96.6 | 99.5 | 96.8 | 97.6 |
| CAGUI-Aug | Without | 97.8 | 98.3 | 98.9 | 98.3 |
| CAGUI-Aug | With | 96.0 | 99.0 | 96.2 | 97.1 |
The mean ASR ranges from 97.1% to 98.9% across these settings, supporting the applicability of ActionPatch beyond the original agent and model configuration.
Revision: We added the clean-scoring protocol and clean-correct evaluation in Appendix G, Table 11, and the additional MobiAgent evaluation in Appendix H, Table 12. We added Appendix I, Additional Target-Argument Configurations, with the new target specifications in Table 13 and their PVR, AMR, and ASR results in Table 14.
reviewer KpuS - part1/4
Thank you for your valuable feedback. We address the raised questions and weaknesses below, and also incorporated them into the revised paper uploaded.
R4-W1 / R4-Q1: Algorithmic contribution beyond Prompt Matching and existing action-aware supervision
Response: Thank you for raising this question. Both prompt matching [1] and MIP [2] optimize overall similarity to the target output without accounting for the importance of key fields to agent task performance. Under small patch budgets, they can therefore achieve high overall similarity while still producing incorrect values in key fields. To address this limitation, we combine a KL divergence loss that preserves the overall output with a weighted cross-entropy (CE) loss that emphasizes key fields. This combination enables targeted control while maintaining a small patch size. Our experiments show improved performance, while end-to-end evaluations further demonstrate the effectiveness of our approach in practical settings.
We agree that our weighting principle is related to GUI-Libra [3], but the key distinction is what each method optimizes. GUI-Libra aims to train the model using action-aware weighting to improve action prediction and grounding. ActionPatch instead keeps the model fixed and optimizes a local input patch, combining execution-field supervision with text-induced soft targets to induce targeted operations.