- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes EVOLVATTACK, a memory aware backdoor attack targeting self-evolving VLM agents. The core idea is to use both a visual trigger and the agent’s evolving memory as joint activation conditions for the backdoor, so that the model behaves normally when encountering the trigger at the beginning of deployment, but the attack is gradually activated and amplified through a positive feedback loop as payload related memories accumulate during interaction. The authors evaluate the attack across multiple VLM backbones, agent pipelines, and memory retrieval mechanisms. The results show that the attack remains nearly dormant at the initial stage, but can achieve a high ASR after a certain number of interactions while largely preserving normal task performance.
Critical strengths
1.The core idea is novel and interesting. Treating the memory accumulated through agent evolution as a cumulative activation condition for the backdoor, rather than directly binding the trigger to the payload, provides an interesting new perspective. It also highlights a genuine security blind spot in self-evolving agents, since static pre-deployment evaluation alone may be insufficient to ensure their safety.
2.The paper provides an experimental evaluation of the core idea. The authors evaluate the attack across three VLM backbones, four agent pipelines, and three memory retrieval mechanisms, and use loss-term ablations to examine the role of each component. The results show that, under the evaluated offline settings, the method can indeed achieve low ASR before evolution and high ASR after evolution while largely preserving task performance, demonstrating that the proposed mechanism is feasible under controlled offline conditions.
Critical weaknesses
1. The current experiments are insufficient to fully support the paper’s core claim about real deployment of self evolving agents. A central claim of the paper is that the backdoor behavior is gradually activated through real interactions after deployment, and the paper repeatedly frames the evaluation in terms of deployment. In practice, however, the experiments rely on a fixed stream of offline replayed screenshots and do not include an end to end run of a real agent. I would like to see at least one closed loop experiment in a genuinely interactive environment to validate this claim.
2. The threat model is quite weak, and the defender’s capabilities are neither strong nor clearly defined. The defense evaluation is limited to memory side defenses, which provides little evidence about practical robustness. From the defender’s perspective, EVOLVATTACK is not fundamentally very different from a conventional parameter implanted visual patch backdoor, since in both cases the deployed model eventually exhibits suspicious behavior. Once such abnormal behavior is observed, existing defenses and detection methods for visual patch backdoors (e.g., runtime detectors such as STRIP) probably provide meaningful protection against EVOLVATTACK, especially because its visual trigger is simply a standard patch trigger. In addition, the stable phase reaches an ASR close to 100%, and the harmful behavior accumulates gradually, which provides a substantial early window for runtime monitoring. It is therefore not obvious that EVOLVATTACK is necessarily more stealthy than an ordinary patch backdoor.
3. Even if the authors’ threat model is fully accepted and the defender is assumed not to perform any detection or mitigation after deployment, the practical relevance of the setting remains limited. EVOLVATTACK is still embedded in the model parameters, while in realistic deployments a third party model is often further fine tuned for downstream tasks rather than being protected only through memory side defenses. It is therefore important to know whether EVOLVATTACK can survive common task specific fine tuning. Moreover, because its stealthiness depends on suppressing the trigger response before sufficient memory accumulation, such behavior may be more vulnerable to being overwritten during fine tuning. The paper does not provide any experiments on this issue.
4. The paper also appears insufficiently polished. The 0.5 percent ASR reported in the Abstract, Introduction, and Conclusion cannot be clearly matched to any result in the experimental section. The claimed 7 to 1.5 to 1.5 data split also does not appear to align clearly with the sample counts reported in the paper. In addition, the ablation discussion uses wording such as “expected to” when describing conclusions that are supposedly already supported by completed experiments. This makes the experimental narrative appear unfinished and reduces confidence in the reported results.
Questions for authors
See weaknesses.
7 comments
Response to W3 — Robustness against Finetuning
To address this concern, we conducted an additional experiment involving clean downstream fine-tuning on Qwen2.5-VL-3B. Importantly, rather than freezing the backdoored adapter and introducing a separate LoRA, we directly continued training the same LoRA adapter that carries EvolvAttack, allowing clean-task gradients to modify the backdoor-bearing parameters.
Specifically, we used LoRA with rank , , dropout , and no trainable bias, covering the language model's attention and MLP layers, the visual encoder's attention layers, and the vision-language merger (40.05M trainable parameters). We fine-tuned on 800 clean GUI grounding/action-prediction samples, including single-element localization and multi-step app operations, for 2 epochs (lr = 1e-5). No triggers or poisoned samples were included.
| Setting | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Mean ASR |
|---|---|---|---|---|---|
| Before fine-tuning | 0.94 | 0.95 | 0.93 | 0.97 | 0.9475 (94.75%) |
| After fine-tuning | 0.94 | 0.99 | 0.95 | 0.98 | 0.9650 (96.50%) |
Results. Under this limited downstream fine-tuning setting (800 clean samples, 2 epochs, lr = 1e-5), EvolvAttack remains effective across all four seeds. The mean stable-stage ASR changes from 94.75% to 96.50%, with no decrease in any evaluated seed. Importantly, we directly fine-tuned the backdoored LoRA adapter rather than freezing it.
These results show that the backdoor can survive moderate clean downstream adaptation, although more extensive fine-tuning may affect its effectiveness.
Response to W4 — Paper Writing
Regarding the ASR discrepancy, the actual ASR is 0.0046 (0.46%), which is rounded to 0.5% in the abstract. Table 1 reports ASR to two decimal places, so the same value is displayed as 0.00. We We have clarified this detail in the revised manuscript. Thanks for pointing it out.
Regarding the dataset split, we construct the train/validation/test pools at a 7:1.5:1.5 ratio from GUI-R1. All evaluation samples are drawn from the held-out test pool, including the 100 unique screenshots used for deployment evaluation.
Thank you for the revisions and additional experiments. I have two remaining questions.
Regarding W3, to clarify, my concern is actually whether EvolvAttack’s dormancy mechanism is preserved after clean fine-tuning, and whether the backdoored model, when directly deployed afterward, can still undergo the full transition from the dormant phase, through the first spark, to the stable attack phase. A high stable-stage ASR alone does not establish this. Could you directly deploy the clean-fine-tuned checkpoints from empty memory, without further backdoor training, and compare their activation curves with those of the original backdoored checkpoints under the same protocol? This comparison should include initial dormancy, first-fire timing, and subsequent amplification.
Regarding W2, I agree that STRIP may fail while the backdoor remains dormant. However, once poisoned memory is held fixed, the memory condition is already satisfied, leaving the visual trigger as the only activation condition. The AND-gate mechanism alone therefore does not explain why STRIP would fail in this setting. Could you clarify how entropy is computed for the VLM outputs, whether memory is fixed across perturbations, and why triggered inputs exhibit higher entropy? The reversed-score AUC of 0.74 also suggests that the conclusion about detectability needs qualification.
We thank the reviewer for spending precious time reviewing the paper and raising these insightful questions. Our responses to each question and weakness are below:
Response to W1 — Real Deployment
We thank the reviewer for this concern. Our original four-framework experiments (§4.4, Appendix E) preserve framework-specific prompting and memory mechanisms but use offline screenshot replay rather than live action execution.
To address this limitation, we conducted an additional closed-loop experiment in a live web application using an AppAgent-based GUI agent. The agent receives live screenshots, predicts actions that Playwright executes, and continuously updates and retrieves memory.
Results. We observe the complete Dormant → Spark → Self-Reinforcing lifecycle. The empty-memory control shows 0% ASR. Across Seeds 1, 42, and 3, the first malicious actions occur at rounds 1, 3, and 30, respectively, with cumulative ASRs of 99%, 97%, and 70% after 100 rounds (Seed 3 still increasing).
Importantly, get_user_info() is actually executed and accesses synthetic user information, rather than merely appearing in offline predictions. Full results and screenshots are provided in Appendix G.
Response to W2 — Threat Model and Defense
Thank you for raising this concern. We agree that the defender’s capabilities and the scope of our stealthiness claim require clearer boundaries.
Defender capabilities and deployment setting. We consider a model-distribution setting in which a platform screens a submitted model before users download and deploy it locally (Hugging Face: Uploading models; Hugging Face: Security). The platform can inspect the model and run test interactions, but it cannot observe users’ subsequent local executions or evolving memory. Different users may integrate the same model into applications with different memory-update mechanisms, which the platform’s screening may not fully cover. This setting separates predeployment screening from monitoring independently operated deployments. We will clarify the platform’s access and testing capabilities in the threat model.
What our stealthiness condition establishes. Our claim concerns behavior before the relevant memory condition develops. As defined in Eq. (1), the model is trained to produce correct outputs even on triggered inputs when memory is absent. Consequently, screening that relies on eliciting malicious outputs under memory-free conditions may fail to expose the backdoor. This does not imply that the model is benign or that all existing detectors will fail. To evaluate whether conventional visual-trigger detection can identify EvolvAttack before memory evolution, we added a STRIP defense evaluation under empty-memory conditions. Under the standard low-entropy detection rule, STRIP achieves an AUC of 0.42 (Appendix J), suggesting that standard STRIP does not reliably detect EvolvAttack under this setting.
Runtime detection and the monitoring window. We agree that, once the memory condition is satisfied, the visual patch and resulting malicious behavior may provide opportunities for runtime detection. Our threat model does not assume that every local deployment are carefull enough includes such monitoring. A model that passes platform screening may therefore provide a false sense of security and be deployed without further checks, allowing malicious behavior to emerge as memory evolves. This possibility motivates studying the gap between predeployment screening and subsequent behavior.