ActionPatch: Distilling Adversarial Instructions into Visual Patches for GUI Agent Hijacking

Mobile GUI agents translate screen content and user requests into executable actions, exposing their control flow to adversarial visual inputs. Precise hijacking through a small image region requires the model's output to specify the intended action and arguments in a format accepted by the agent's parser. We present , a method that distills text-induced adversarial behavior into compact visual patches, each targeting a GUI action and its arguments. With the vision-language model frozen, the method combines teacher-distribution matching with weighted supervision on execution-critical tokens, emphasizing action and argument fidelity while maintaining supervision over the full response. Each patch is optimized across benign requests and screenshots for reuse across interaction contexts. Independently optimized patches are composed into attack chains by exposing them at successive interaction stages. Experiments on Mobile-Agent-v3.5 show that small visual patches reliably cause the agent to execute target actions with the specified arguments across held-out benign requests. Ablation studies show that combining distribution matching with execution-critical token supervision improves attack success over either objective alone under matched patch budgets. A series of successful attack-chain case studies demonstrates how atomic patches achieve multi-step malicious goals and amplify the impact of individual action hijacks.