acceptodds
Under review as a conference paper at ICLR 2027

EvolvAttack: Backdooring Self-Evolving VLM Agents

Abstract

Self-evolving mechanisms let vision-language model (VLM) agents continually adapt through evolving memory, but they also allow malicious behavior to emerge only after deployment. We introduce a new attack surface, the self-evolving backdoor, whose payload stays dormant in the initial agent and is progressively activated as the agent evolves. Unlike conventional backdoors, which can be exposed by testing triggered inputs on a static model, our attack requires both a visual trigger and a sufficiently evolved memory state; static scanning with clean memory thus yields near-zero attack success and may deem the agent benign. We realize this through memory-aware backdoor training, which couples trigger activation with the agent's evolving memory while preserving benign behavior unless the trigger co-occurs with payload-related memory. Across three VLM backbones, four agent pipelines, and recency-, semantic-, and reward-based memory evolution, the attack preserves normal functionality and stays nearly inactive before evolution (0.5% attack success rate), yet exceeds 95% attack success within 100 interactions, revealing a stealthy threat unique to self-evolving agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Threat Model, VLM Agents, Backdoor Attacks, Agent Memory, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6BKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes EVOLVATTACK, a memory aware backdoor attack targeting self-evolving VLM agents. The core idea is to use both a visual trigger and the agent’s evolving memory as joint activation conditions for the backdoor, so that the model behaves normally when encountering the trigger at the beginning of deployment, but the attack is gradually activated and amplified through a positive feedback loop as payload related memories accumulate during interaction. The authors evaluate the attack across multiple VLM backbones, agent pipelines, and memory retrieval mechanisms. The results show that the attack remains nearly dormant at the initial stage, but can achieve a high ASR after a certain number of interactions while largely preserving normal task performance.

Critical strengths

1.The core idea is novel and interesting. Treating the memory accumulated through agent evolution as a cumulative activation condition for the backdoor, rather than directly binding the trigger to the payload, provides an interesting new perspective. It also highlights a genuine security blind spot in self-evolving agents, since static pre-deployment evaluation alone may be insufficient to ensure their safety.

2.The paper provides an experimental evaluation of the core idea. The authors evaluate the attack across three VLM backbones, four agent pipelines, and three memory retrieval mechanisms, and use loss-term ablations to examine the role of each component. The results show that, under the evaluated offline settings, the method can indeed achieve low ASR before evolution and high ASR after evolution while largely preserving task performance, demonstrating that the proposed mechanism is feasible under controlled offline conditions.

Critical weaknesses

1. The current experiments are insufficient to fully support the paper’s core claim about real deployment of self evolving agents. A central claim of the paper is that the backdoor behavior is gradually activated through real interactions after deployment, and the paper repeatedly frames the evaluation in terms of deployment. In practice, however, the experiments rely on a fixed stream of offline replayed screenshots and do not include an end to end run of a real agent. I would like to see at least one closed loop experiment in a genuinely interactive environment to validate this claim.

2. The threat model is quite weak, and the defender’s capabilities are neither strong nor clearly defined. The defense evaluation is limited to memory side defenses, which provides little evidence about practical robustness. From the defender’s perspective, EVOLVATTACK is not fundamentally very different from a conventional parameter implanted visual patch backdoor, since in both cases the deployed model eventually exhibits suspicious behavior. Once such abnormal behavior is observed, existing defenses and detection methods for visual patch backdoors (e.g., runtime detectors such as STRIP) probably provide meaningful protection against EVOLVATTACK, especially because its visual trigger is simply a standard patch trigger. In addition, the stable phase reaches an ASR close to 100%, and the harmful behavior accumulates gradually, which provides a substantial early window for runtime monitoring. It is therefore not obvious that EVOLVATTACK is necessarily more stealthy than an ordinary patch backdoor.

3. Even if the authors’ threat model is fully accepted and the defender is assumed not to perform any detection or mitigation after deployment, the practical relevance of the setting remains limited. EVOLVATTACK is still embedded in the model parameters, while in realistic deployments a third party model is often further fine tuned for downstream tasks rather than being protected only through memory side defenses. It is therefore important to know whether EVOLVATTACK can survive common task specific fine tuning. Moreover, because its stealthiness depends on suppressing the trigger response before sufficient memory accumulation, such behavior may be more vulnerable to being overwritten during fine tuning. The paper does not provide any experiments on this issue.

4. The paper also appears insufficiently polished. The 0.5 percent ASR reported in the Abstract, Introduction, and Conclusion cannot be clearly matched to any result in the experimental section. The claimed 7 to 1.5 to 1.5 data split also does not appear to align clearly with the sample counts reported in the paper. In addition, the ablation discussion uses wording such as “expected to” when describing conclusions that are supposedly already supported by completed experiments. This makes the experimental narrative appear unfinished and reduces confidence in the reported results.

Questions for authors

See weaknesses.

7 comments

We thank the reviewer for spending precious time reviewing the paper and raising these insightful questions. Our responses to each question and weakness are below:

Response to W1 — Real Deployment

We thank the reviewer for this concern. Our original four-framework experiments (§4.4, Appendix E) preserve framework-specific prompting and memory mechanisms but use offline screenshot replay rather than live action execution.

To address this limitation, we conducted an additional closed-loop experiment in a live web application using an AppAgent-based GUI agent. The agent receives live screenshots, predicts actions that Playwright executes, and continuously updates and retrieves memory.

Results. We observe the complete Dormant → Spark → Self-Reinforcing lifecycle. The empty-memory control shows 0% ASR. Across Seeds 1, 42, and 3, the first malicious actions occur at rounds 1, 3, and 30, respectively, with cumulative ASRs of 99%, 97%, and 70% after 100 rounds (Seed 3 still increasing).

Importantly, get_user_info() is actually executed and accesses synthetic user information, rather than merely appearing in offline predictions. Full results and screenshots are provided in Appendix G.

Response to W2 — Threat Model and Defense

Thank you for raising this concern. We agree that the defender’s capabilities and the scope of our stealthiness claim require clearer boundaries.

Defender capabilities and deployment setting. We consider a model-distribution setting in which a platform screens a submitted model before users download and deploy it locally (Hugging Face: Uploading models; Hugging Face: Security). The platform can inspect the model and run test interactions, but it cannot observe users’ subsequent local executions or evolving memory. Different users may integrate the same model into applications with different memory-update mechanisms, which the platform’s screening may not fully cover. This setting separates predeployment screening from monitoring independently operated deployments. We will clarify the platform’s access and testing capabilities in the threat model.

What our stealthiness condition establishes. Our claim concerns behavior before the relevant memory condition develops. As defined in Eq. (1), the model is trained to produce correct outputs even on triggered inputs when memory is absent. Consequently, screening that relies on eliciting malicious outputs under memory-free conditions may fail to expose the backdoor. This does not imply that the model is benign or that all existing detectors will fail. To evaluate whether conventional visual-trigger detection can identify EvolvAttack before memory evolution, we added a STRIP defense evaluation under empty-memory conditions. Under the standard low-entropy detection rule, STRIP achieves an AUC of 0.42 (Appendix J), suggesting that standard STRIP does not reliably detect EvolvAttack under this setting.

Runtime detection and the monitoring window. We agree that, once the memory condition is satisfied, the visual patch and resulting malicious behavior may provide opportunities for runtime detection. Our threat model does not assume that every local deployment are carefull enough includes such monitoring. A model that passes platform screening may therefore provide a false sense of security and be deployed without further checks, allowing malicious behavior to emerge as memory evolves. This possibility motivates studying the gap between predeployment screening and subsequent behavior.

Response to W3 — Robustness against Finetuning

To address this concern, we conducted an additional experiment involving clean downstream fine-tuning on Qwen2.5-VL-3B. Importantly, rather than freezing the backdoored adapter and introducing a separate LoRA, we directly continued training the same LoRA adapter that carries EvolvAttack, allowing clean-task gradients to modify the backdoor-bearing parameters.

Specifically, we used LoRA with rank r=16r=16, α=32\alpha=32, dropout =0.05=0.05, and no trainable bias, covering the language model's attention and MLP layers, the visual encoder's attention layers, and the vision-language merger (40.05M trainable parameters). We fine-tuned on 800 clean GUI grounding/action-prediction samples, including single-element localization and multi-step app operations, for 2 epochs (lr = 1e-5). No triggers or poisoned samples were included.

SettingSeed 1Seed 2Seed 3Seed 4Mean ASR
Before fine-tuning0.940.950.930.970.9475 (94.75%)
After fine-tuning0.940.990.950.980.9650 (96.50%)

Results. Under this limited downstream fine-tuning setting (800 clean samples, 2 epochs, lr = 1e-5), EvolvAttack remains effective across all four seeds. The mean stable-stage ASR changes from 94.75% to 96.50%, with no decrease in any evaluated seed. Importantly, we directly fine-tuned the backdoored LoRA adapter rather than freezing it.

These results show that the backdoor can survive moderate clean downstream adaptation, although more extensive fine-tuning may affect its effectiveness.

Response to W4 — Paper Writing

Regarding the ASR discrepancy, the actual ASR is 0.0046 (0.46%), which is rounded to 0.5% in the abstract. Table 1 reports ASR to two decimal places, so the same value is displayed as 0.00. We We have clarified this detail in the revised manuscript. Thanks for pointing it out.

Regarding the dataset split, we construct the train/validation/test pools at a 7:1.5:1.5 ratio from GUI-R1. All evaluation samples are drawn from the held-out test pool, including the 100 unique screenshots used for deployment evaluation.

Thank you for the revisions and additional experiments. I have two remaining questions.

Regarding W3, to clarify, my concern is actually whether EvolvAttack’s dormancy mechanism is preserved after clean fine-tuning, and whether the backdoored model, when directly deployed afterward, can still undergo the full transition from the dormant phase, through the first spark, to the stable attack phase. A high stable-stage ASR alone does not establish this. Could you directly deploy the clean-fine-tuned checkpoints from empty memory, without further backdoor training, and compare their activation curves with those of the original backdoored checkpoints under the same protocol? This comparison should include initial dormancy, first-fire timing, and subsequent amplification.

Regarding W2, I agree that STRIP may fail while the backdoor remains dormant. However, once poisoned memory is held fixed, the memory condition is already satisfied, leaving the visual trigger as the only activation condition. The AND-gate mechanism alone therefore does not explain why STRIP would fail in this setting. Could you clarify how entropy is computed for the VLM outputs, whether memory is fixed across perturbations, and why triggered inputs exhibit higher entropy? The reversed-score AUC of 0.74 also suggests that the conclusion about detectability needs qualification.

Reviewer_3V4VOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies memory-dependent backdoors in self-evolving VLM agents. EvolvAttack trains a compromised model to remain largely benign on triggered inputs with clean or empty memory, while producing an attacker-specified output when the visual trigger co-occurs with payload-related memory. Occasional initial emissions seed the memory, and subsequent write-back and retrieval amplify activation without further parameter updates or direct attacker access to deployment memory.

The evaluation covers three VLM backbones, three retrieval mechanisms, and integrations with four GUI-agent frameworks. It reports low initial activation followed by high attack success, supported by training-objective ablations and memory-side defense experiments. The framework evaluations use offline screenshot replay, and attack success measures emission of a placeholder payload rather than execution of a harmful operation.

Critical strengths

The paper identifies a consequential limitation of initialization-only security evaluation. A model can appear benign under clean-memory testing while its own outputs progressively create conditions for later malicious behavior. The distinctive contribution is the endogenous accumulation and regeneration of attack-related memory, rather than conditional activation alone. This provides a useful security perspective on agents whose parameters remain fixed while their memory evolves.

The experiments provide meaningful evidence for the proposed activation mechanism. The evaluation spans different backbones and memory organizations, while the ablations separate the roles of payload learning, initial dormancy, and trigger-dependent activation. The memory-sanitization experiment further illustrates why removing contaminated entries may provide only temporary relief when the compromised model can regenerate them. Although the evaluation remains controlled, these results support the central mechanism and motivate trajectory-level security testing.

Critical weaknesses

The practical validation is narrower than the deployment framing suggests. The four framework integrations preserve prompting and memory components but replace environment execution with a fixed screenshot stream. The payload is logged rather than executed. Consequently, the results establish memory-conditioned output generation and amplification, but do not establish successful compromise of a closed-loop GUI agent. Real action outcomes could affect subsequent observations, rewards, reflections, and whether malicious experiences are retained. The paper discloses this setup, but should more consistently distinguish the demonstrated mechanism from broader claims about deployed-agent behavior.

The attacker assumptions and scope of stealth require clearer qualification. The threat model emphasizes poisoning a small subset of training data, whereas the method uses simulated memory states, structured conditional supervision, and balanced clean/triggered sampling. The required control over training and the effective poisoning budget are therefore not sufficiently clear. Moreover, low activation under empty or clean memory does not establish that static evaluation with appropriately varied memory contexts would also fail. These issues limit the strength of the general security claims, although they do not negate the controlled demonstration. Overall, I favor weak acceptance for the useful mechanism and security insight, with these qualifications.

Questions for authors
  1. Attacker capabilities and budget: Does the attacker control the model's training procedure, or only contribute poisoned examples to an otherwise standard fine-tuning pipeline? How do simulated memory variants and the equal clean/triggered sampling affect the effective poisoning budget relative to the stated 15% of training samples?

  2. Memory retention under deployment: In the offline framework experiments, how are rewards, reflection decisions, and memory-write gates determined when predicted actions are not executed? What evidence shows that payload-bearing outputs remain eligible for retention under the frameworks' normal success- or reflection-conditioned updates?

  3. Scope of static-test evasion: Have you evaluated static triggered inputs with payload-bearing or systematically varied memory contexts? Please clarify whether the evidence supports evasion of clean-memory screening specifically, or a broader claim that detecting the backdoor requires replaying a long interaction trajectory.

4 comments

We thank the reviewer for spending precious time reviewing this paper and raising insightful questions, also thanks for supporting. Our responses to each question and weakness are below:

Q1: Attacker’s Capability

Following existing work [Gu et al., 2017], we assume the attacker to control the training procedure, this assumption is realistic in model supply chain settings, and widely used in existing work.

We clarify that 15% poisoning ratio (375/2,498 samples) and 1:1 clean-to-triggered sampling refer to two different quantities.

The 15% refers to the proportion of distinct poisoned examples in the training dataset. During training, we use a 1:1 balanced sampling strategy to facilitate optimization. This reuses existing poisoned examples without introducing additional distinct poisoned samples. Therefore, the reported 15% dataset-level poisoning ratio remains correct, although triggered examples account for 50% of training-time samples.

Importantly, the 15% ratio is a training-data statistic, not our definition of stealthiness, and it only applies to the training phase. Our stealthiness objective is defined in §3.2 and evaluated in Table 1 (§4.2), which shows low pre-evolution ASR while maintaining high post-evolution ASR.

To further avoid confusion, trigger density is a distinct deployment-time variable. It controls how frequently the visual trigger appears across interaction rounds. As shown in Appendix D, Fig. 8, higher trigger density leads to faster attack amplification in our evaluated settings, while lower trigger density delays the increase in rolling ASR.

Q2: Memory retention under deployment

We appreciate this question and clarify that the paper presents two different evaluation settings, which serve different purposes.

(1) Controlled Memory-Gate Study (§4.2, Table 1). The memory gates mentioned by the reviewer are used in this controlled study. Here, we evaluate the backdoored VLM with representative memory write/retrieval rules, separately from the full agent pipelines. This allows us to study more clear how different memory mechanisms affect the attack without involving other framework-specific components, such as prompt formats and execution procedures.

Memory mechanismMemory-write Gate
Recency-basedAppend each step and retain the most recent k entries (FIFO).
SemanticApply a novelty gate; reject an entry when maximum cosine similarity is at least 0.90.
Reward/reflection-basedRetain a reflection when the outcome belongs to the designated wrong/target-hit categories.

(2) Real agent-pipeline deployment (Section 4.4). However, beyond the controlled study, §4.4 runs four fully real agent pipelines, each using the framework’s own native memory write rule — we did not substitute any gate of our own in this session.

FrameworkMemory component considered in the integration
AppAgentBounded FIFO action-history buffer (memory size = 5)
SeeActAppends previous actions
Mobile-Agent v3.5Append per-step reflection
Agent-SEpisodic/subtask experience store with semantic recall

Q3: Scope of Static Evaluation

(1) Evaluation with Payload-Bearing Memory. As expected from our AND-gate design, triggered inputs paired with payload-bearing memory can activate the backdoor. We do not claim stealthiness under this condition, this is our intended attack scenario. However, constructing such a memory context requires knowledge of the attack payload, which may not be available to a black-box evaluator.

(2) Evaluation of random memory context with static screening Regarding the scope of static screening with varied memory contexts, we supplemented the evaluation with a out-of-distribution memory experiment. For each seed, we constructed a static memory containing K=20K=20 unrelated entries sampled from WikiText-103, and held that memory fixed during 100 trigger-bearing probes. No memory write-back or retrieval evolution occurred. At the deployed Spark rate ρspark=0.10\rho_{\mathrm{spark}}=0.10, the results were:

Memory conditionSeeds × probesPooled activation ratePer-seed activation rates
Frozen unrelated memory (K=20K=20)3 × 100 = 3001.67% (5/300)3%, 0%, 2%

The per-seed mean ± sample standard deviation was 1.67 ± 1.53 percentage points. This result provides additional evidence that the model's activation remains low under this particular non-evolving, out-of-distribution memory condition.This also tells that even audit using random memory text and static screening, our method shows robustness.

(3) Is Long-Horizon Interaction Necessary for Detection? In real agent deployment, detection depends on trigger density and memory update rules. Our 51-seed Agent-S evaluation shows that reaching a 20% rolling ASR threshold requires an average of 25 rounds under 100% trigger density (§4.6, Figure 5). When the attacker gradually increases trigger density (10% → 25% → 50% → 75% → 100%), rolling ASR reaches approximately 50% only around round 100 (Appendix D, Fig. 8). These results suggest that static screening alone may be insufficient, while lower trigger density can further delay attack amplification.

Response to W1 — Gap Between Deployment Claims and Practical Validation

We thank the reviewer for this concern. Our original four-framework experiments (§4.4, Appendix E) preserve framework-specific prompting and memory mechanisms but use offline screenshot replay rather than live action execution.

To address this limitation, we conducted an additional closed-loop experiment in a live web application using an AppAgent-based GUI agent. The agent receives live screenshots, predicts actions that Playwright executes, and continuously updates and retrieves memory.

Results. We observe the complete Dormant → Spark → Self-Reinforcing lifecycle. The empty-memory control shows 0% ASR. Across Seeds 1, 42, and 3, the first malicious actions occur at rounds 1, 3, and 30, respectively, with cumulative ASRs of 99%, 97%, and 70% after 100 rounds (Seed 3 still increasing).

Importantly, get_user_info() is actually executed and accesses synthetic user information, rather than merely appearing in offline predictions. Full results and screenshots are provided in Appendix G.

Response to W2 — Threat Model, Poisoning Budget, and Limits of Static Stealth

We thank the reviewer for these concerns, which we have addressed in our responses to Q1 and Q3.

Regarding attacker capabilities and poisoning budget, please see our response to Q1, where we clarify the attacker's control over training, the role of simulated memory states, and the difference between the 15% dataset-level poisoning ratio and 1:1 training-time sampling.

Regarding the scope of static-test evasion, please see our response to Q3, where we report additional static screening results with unrelated memory contexts (1.67% ASR, 5/300 probes) and clarify the limits of our stealthiness claim.

Thank you for your effort for your rebuttal, I will keep the score 3 and wish the paper be accepted.

Thank you very much for your support and encouragement throughout the review process. We sincerely appreciate your constructive feedback, valuable suggestions, and the time you have taken to review our work.

Reviewer_p6agOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates delayed-activation backdoor attacks against VLM agents with dynamic memory update mechanisms. Unlike traditional backdoors, which typically activate immediately when a specific trigger appears, this paper proposes EvolvAttack, which aims to maintain a low backdoor activation rate during the initial deployment stage of an agent and gradually activate the backdoor as memory is continuously updated through interactions. Specifically, the authors poison a portion of the fine-tuning data and employ a memory-aware backdoor training method to associate visual triggers with memory contexts containing malicious payloads. They also design four loss functions to control normal task performance, backdoor activation, initial dormancy, and conditional triggering, respectively. Meanwhile, the authors introduce a Spark mechanism that uses low-probability malicious label flipping during training, allowing the model to occasionally produce its first malicious output even before malicious memories have accumulated. This output is subsequently written into memory by the agent and retrieved during later interactions, forming a self-reinforcing attack process without requiring any model parameter updates after deployment. The experiments cover three VLMs, four agent frameworks, and three types of memory retrieval mechanisms. The authors report that the attack achieves a success rate of approximately 0.5% in the initial stage, which increases to over 95% within 100 interaction rounds, while also demonstrating some resistance to memory sanitization and dilution defenses. It should be noted that the experiments primarily use offline GUI screenshot replay, measure attack success based on the generation of malicious action strings, and do not verify the actual execution of malicious operations in a real interactive environment.

Critical strengths

The paper focuses on a relatively new attack surface: the security behavior of an agent may change as long-term memory accumulates, meaning that static security evaluations conducted only at the initial stage may not fully reflect the security risks during its subsequent operation. The paper concretizes this issue into an attack scenario involving the interaction between model-parameter backdoors and dynamic memory, providing a specific case for studying security risks throughout the agent lifecycle.

Critical weaknesses
  1. The technical novelty has not been sufficiently distinguished or validated. The core mechanism of the paper can be decomposed into existing multi-condition backdoors, low-probability malicious label supervision, and the self-reinforcing process of memory poisoning. Although combining these mechanisms forms a specific attack approach, the paper lacks direct comparisons with relevant methods such as an adapted Dual-Key Backdoor. Therefore, it remains unclear what additional technical contributions the proposed memory-aware training method offers compared with a combination of existing conditional backdoors and memory feedback mechanisms.
  2. The central stealthiness claim lacks sufficient experimental support. The self-initiating process of EvolvAttack relies on the Spark mechanism, which allows the model to produce its first malicious output with a low probability before the memory has been contaminated. Therefore, there is a trade-off between the attack's actual stealthiness and its subsequent activation efficiency that needs to be validated. However, the paper does not systematically investigate how different Spark parameters affect the initial activation probability and long-term attack effectiveness, nor does it include experiments on security evaluation procedures involving multi-round interactions and memory write-back. The current results mainly demonstrate a low ASR under empty-memory conditions and an increasing attack success rate during subsequent interactions. However, they are insufficient to directly demonstrate that the method can remain stealthy under more comprehensive pre-deployment security evaluations while activating promptly and reliably after actual deployment.
  3. The experimental setup limits the applicability of the conclusions regarding attack effectiveness. Although the authors conduct experiments across four agent frameworks, these experiments uniformly modify the frameworks' native action interfaces and output formats and use offline GUI screenshot replay rather than closed-loop interactions in real environments. Therefore, the existing experiments support the attack's effectiveness at the model-output level but are insufficient to demonstrate its effectiveness within the execution pipeline of real agents.
Questions for authors
  1. How is the Self-Evolving Agent studied in this paper defined? Does the attack rely on properties unique to self-evolving mechanisms? This question directly concerns the paper's research motivation and threat model. Based on the current method description, the paper mainly characterizes an agent's self-evolution as the continuous updating of external memory, while the model parameters remain unchanged after deployment. However, this mechanism is also widely present in ordinary agents with memory mechanisms. For example, a RAG agent that continuously writes historical experiences into an external knowledge base could theoretically also satisfy the attack conditions considered in this paper. Therefore, I would like the authors to further clarify the distinction between self-evolving agents and general memory-augmented agents in this paper, and explain which specific properties of self-evolving mechanisms the proposed attack actually exploits, rather than merely relying on continuously updated external context.
  2. What is the fundamental difference in technical mechanism between EvolvAttack and existing dual-condition backdoor attacks? The backdoor activation in this paper depends on the simultaneous presence of a visual trigger and memory containing malicious content, which is relatively close to the basic idea of existing multi-condition backdoors. If a conventional dual-key backdoor were used, with the visual patch as the first triggering condition and specific text or patterns in memory as the second triggering condition, combined with a low-probability initial malicious output mechanism and the agent's own memory write-back process, could it achieve similar delayed activation and self-reinforcing effects? I would like the authors to demonstrate through more direct comparative experiments what irreplaceable role the memory-aware training method in EvolvAttack plays compared with such a combined approach.
  3. Could the authors define and consistently describe the stealthiness of the backdoor more rigorously throughout the paper? The paper emphasizes that EvolvAttack can remain dormant during the initial deployment stage and is unlikely to activate even in the presence of a visual trigger. However, the Spark mechanism requires the model to have a non-zero probability of producing malicious outputs under clean memory conditions. Otherwise, the attack cannot autonomously initiate the first memory contamination and consequently cannot enter the subsequent self-reinforcing stage. I would like the authors to provide a consistent definition of stealthiness across the threat model, method formulation, and experimental analysis, and clarify whether the claimed stealthiness refers to strict conditional dormancy or probabilistic stealthiness under limited testing conditions.
  4. Could the authors systematically validate the relationship between Spark parameters, attack stealthiness, first activation time, and long-term effectiveness? Spark is the key mechanism that allows EvolvAttack to autonomously initiate the memory poisoning process, and its design simultaneously affects two objectives. On the one hand, initial malicious outputs must be sufficiently rare to reduce the likelihood of detection before deployment. On the other hand, the first malicious output must occur within a reasonable number of interaction rounds, otherwise, the backdoor may fail to enter the stable attack phase for a long time. However, the paper mainly adopts a fixed Spark setting and does not sufficiently demonstrate how different training-time label-flipping rates affect the actual first activation probability, the distribution of first fire round, and long-term ASR. Therefore, I would like the authors to supplement experiments under different Spark parameters and report the distribution of first activation times (interaction rounds) across multiple independent runs. I also suggest simulating pre-deployment security evaluations with different testing budgets, including multi-round tests that allow memory to be continuously updated, and analyzing the probability of detecting the attack. This experiment is particularly important and can directly address the central question of the paper: Can EvolvAttack achieve a reproducible balance between sufficiently low initial detectability and a reasonable subsequent activation speed? The current paper separately demonstrates a low initial ASR and a high post-evolution ASR, but has not sufficiently established the relationship between them, nor provided a systematic evaluation of this trade-off.
7 comments

We thank the reviewer for the insightful questions. Our responses follow:

Q1 Definition of Self-evolving Agents and How Self-Evolving Mechanisms are Exploited

We follow existing work Gao et al., 2026 and define self-evolving agents as the agents that  can learn from an agent's own interaction trajectories. Self-evolution can update model parameters, skills, or memory. We focus on memory updates, a widely used mechanism formally defined in Sec. 2.

EvolvAttack differs from RAG-based attacks in two ways:

(1) RAG-based attacks assume the attacker can pollute the knowledge base (Chen et al., 2024; Zou et al., 2025). However, in our threat model, the attacker CANNOT directly modify the agent's memory, which is generated from its historical trajectories through a closed-loop update process. EvolvAttack therefore uses self-reinforcing memory poisoning, making the agent produce content that contaminates its own memory during evolution.

(2) Unlike the RAG poisoning attacks considered here (Chen et al., 2024; Zou et al., 2025), EvolvAttack explicitly accounts for the temporal dependency between memory updates and backdoor activation. Initially, when memory is empty, the backdoored model is trained to behave normally even on triggered inputs. As the agent incorporates its own interaction trajectories into memory, the accumulated content enables subsequent backdoor activation. The four training objectives in Eqs. (2)–(5) are designed to support this progression. An agent that does not update memory from its own trajectories lacks the feedback loop required for this process. Thus, EvolvAttack targets the interaction between trajectory-based memory updates and future behavior, rather than merely the presence of external memory. Also, apply EvolvAttack to non self-evolving agents will not achieve the attack purpose.

Q2 Difference with Dual-Key Attack

(1) Static Multimodal Triggers vs. Evolving Memory Conditions. The Dual-Key and Composite Backdoor studies considered here (Walmer et al., 2022; Huang et al., 2024) use predefined combinations of input triggers. Although these triggers span visual and textual modalities, activation depends on their presence in the current input, rather than on a sequence of agent interactions and memory updates. These studies therefore do not examine how an agent’s own accumulated experiences can progressively satisfy a backdoor activation condition. In contrast, EvolvAttack targets this temporal dependency: the visual trigger alone is insufficient initially, while memory accumulated through the agent’s interactions provides the second condition for activation.

(2) Different Runtime Attacker Capabilities. The multimodal attacks considered here (Walmer et al., 2022; Huang et al., 2024) require the attacker to supply both visual and textual triggers at runtime. In contrast, EvolvAttack assumes runtime control only over the visual input, without direct access to modify the agent’s memory. The memory content needed for activation is generated and stored through the agent’s own interaction and memory-update processes, rather than directly inserted by the attacker. This distinction reduces the required runtime control over input modalities and memory.

(3) Empirical Evidence of Dynamic Attack. Existing Dual-Key and Composite Backdoor attacks report high ASRs when all required triggers are provided: 91.8% for Dual-Key (Walmer et al., 2022, Table 2) and 100% for Composite Backdoor (Huang et al., 2024, Emotion dataset). In contrast, EvolvAttack achieves only 0.33% ASR with empty memory, even when the visual trigger is present, while reaching 95% ASR after memory evolution. These results illustrate the different activation settings: conventional attacks rely on predefined input triggers, whereas EvolvAttack uses memory accumulated through the agent's own interactions as the second activation condition.

Q3 Definition of Stealthiness

We define stealthiness in Eq. (1) as

Fθ(It⊕τ,qt,NULL)=yt.F_{\theta}\left(I_t \oplus \tau, q_t, \mathrm{NULL}\right)=y_t.

where NULL\mathrm{NULL} denotes the absence of memory and yty_t is the correct output. This condition captures static security screening before deployment, such as checks performed by a model-hosting platform: before the model accumulates interaction memory, it should produce correct outputs even when the visual trigger is present.

The Spark mechanism is consistent with Eq. (1), which constrains only the empty-memory (NULL) regime. Under NULL, EvolvAttack returns the benign and correct action yty_t even with the trigger present (empty-memory ASR =0.003= 0.003), so pre-deployment screening cannot flag it. The Spark is a post-deployment event: once the agent has accumulated a few rounds of clean memory, it emits the first malicious output $y_{\mathrm{target}} = y_t + $ get_user_info() (§3.4, Appendix B) and writes this payload-bearing entry — the second key of the AND-gate — into the memory pool. How often the Spark fires is evaluated across Spark rates below.

Q3 Detailed Evaluation of Sparking

How different spark effects pre-deployment ASR and distribution of FFR

We evaluate pre-deployment ASR across different Spark rates using three independent seeds per setting:

Spark rate ρspark\rho_{\mathrm{spark}}Average Pre-deployment ASR (seed=42,43,44)
0.000.0%
0.050.3%
0.10 (default)0.3%
0.150.7%
0.200.7%

Across all Spark rates, the pre-deployment ASR remains below 1% under empty-memory screening. These results show that varying spark from 0 to 0.20 has little impact on pre-deployment ASR, which remains below 1% across all tested settings.

We next examine how these Spark rates affect first-fire time and long-term attack effectiveness under multi-round interaction with continuous memory updates. We conducted an additional Spark-rate sweep with 11 independent seeds per setting, measuring the first-fire round (FFR, first malicious output after several visual triggers) over a 100-round interaction test. The table reports median, linearly interpolated IQR, observed range, and steady-state cascade ASR averaged over the final 10 rounds (what we call "stable attack phase" in the paper).

Spark rate ρspark\rho_{\mathrm{spark}}Median FFRIQR (Q1–Q3)Observed FFR rangeLast-10-round ASR
0.00——No fire in 100 rounds (11/11)—
0.0554–102–2487.3%
0.1043–6.52–14100.0%
0.1543.5–72–983.6%
0.2044–6.53–1652.7%

Our understanding of this table is:

  1. Spark is necessary for autonomous bootstrapping: at ρspark=0\rho_{\mathrm{spark}}=0, none of the 11 runs fired within 100 rounds. For every positive Spark rate, all 11 seeds eventually fired within at most 24 rounds, with a median FFR of 4–5 rounds. However, larger Spark rates do not monotonically improve long-term effectiveness. 

  2. This supports the choice used in the main experiments. The stable-stage ASR is highest at ρspark=0.10\rho_{\mathrm{spark}}=0.10, where all 11 runs reached 100% last-10-round ASR (zero observed cross-seed variation).  We additionally visualize the per-seed first-fire rounds, together with the median and interquartile range, in the revised manuscript (Appendix F).

Q4 Detection Cost Under Pre- and Post-Deployment Auditing

We added section 4.6 into the revised paper to address this more clear. We also present the core idea and results here:

1. Pre-deployment Audit Budget

We evaluate two static screening conditions:

(A) Empty memory. With no memory updates, ASR is only 0.33% (1/300 probes) across three seeds (42, 43, 44), as reported in Q3.

(B) Random memory. With frozen random memory contexts, ASR remains low at 1.67% (5/300 probes) across three seeds.

Both conditions demonstrate near-zero attack activation under static screening, even when the visual trigger is present.

2. Post-deployment Audit Budget

We further evaluate detection in the Agent-S pipeline across 51 independent seeds (0–50) at spark rate 0.10.

(A) First Fire Round (FFR). All 51 seeds produce a first malicious output within 200 rounds. The median FFR is 10 rounds (IQR: 7.0–18.8), compared with 4 rounds in the simplified memory-pool experiment. This difference reflects the impact of real agent memory-update rules on activation time.

(B) Repeated Malicious Behavior. A single malicious output may be insufficient to confirm a backdoor. We therefore measure when rolling ASR first reaches 20% (window = 10). Agent-S reaches this threshold after a median of 19 rounds (mean: 25; IQR: 13–34) under 100% trigger density.

(C) Lower Trigger Density. The above experiments use 100% trigger density. In practice, an attacker can delay attack amplification by gradually increasing trigger density (10%, 25%, 50%, 75%, 100%). Under this strategy, rolling ASR reaches approximately 50% around round 100 (Appendix D, Fig. 8).

These additional experiments characterize different auditing requirements before and after deployment. We have consolidated the results in Appendix H.

Response to W1 — Technical Novelty and Missing Direct Baselines

Please see Response to Q2, which clarifies how the second activation condition is generated through the agent's own memory-feedback process and distinguishes this from an externally supplied dual-key input.

Response to W2 — Evidence for Stealthiness and the Spark Trade-off

Please see Responses to Q3 and Q4 for our operational definition of probabilistic stealthiness, the multi-seed Spark sweep, first-fire distributions, and the additional pre-deployment screening results.

Response to W3 — Limits of Offline Replay for Real-Agent Execution

We thank the reviewer for this concern. Our original four-framework experiments (§4.4, Appendix E) preserve framework-specific prompting and memory mechanisms but use offline screenshot replay rather than live action execution.

To address this limitation, we conducted an additional closed-loop experiment in a live web application using an AppAgent-based GUI agent. The agent receives live screenshots, predicts actions that Playwright executes, and continuously updates and retrieves memory.

Results. We observe the complete Dormant → Spark → Self-Reinforcing lifecycle. The empty-memory control shows 0% ASR. Across Seeds 1, 42, and 3, the first malicious actions occur at rounds 1, 3, and 30, respectively, with cumulative ASRs of 99%, 97%, and 70% after 100 rounds (Seed 3 still increasing).

Importantly, get_user_info() is actually executed and accesses synthetic user information, rather than merely appearing in offline predictions. Full results and screenshots are provided in Appendix G.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank all reviewers for their constructive feedback. In addition to minor wording and presentation fixes, the revised PDF adds the following new sections and experiments. Detailed results and point-by-point discussion are provided in the individual responses below.

  1. Live closed-loop deployment (§4.4, Appendix G): a demonstration of the attack in a real executable web-app environment.

  2. Pre- and post-deployment detection budgets (new §4.6, Appendix H): an analysis of the cost of detecting the attack before versus after deployment.

  3. Robustness to downstream fine-tuning (Appendix I): an evaluation of the attack when the deployed model is further fine-tuned on a downstream task.

  4. STRIP defense evaluation (§4.7, Appendix J): an evaluation of the attack against the entropy-based STRIP backdoor detector, as an representative conventional visual trigger backdoor detecting method.

  5. Spark-rate ablation (Table 3 in §4.5, Appendix F): an ablation of the spark-rate hyperparameter that governs trigger activation.

We are happy to elaborate on any of these in the individual responses below. All revisions and additions are highlighted in blue color.

Dear Reviewer 6BKK,

Thank you for your thoughtful feedback and for highlighting the novelty of treating the agent’s evolving memory as a cumulative activation condition.

In our rebuttal, we addressed your main concerns with new experiments (marked in blue in the updated PDF): (1) a closed-loop experiment in a live GUI environment confirming the gradual, memory-driven amplification beyond offline replay; (2) an evaluation showing STRIP fails to flag the attack in the pre-deployment regime, along with an audit-budget analysis showing detection requires sustained trajectory-level monitoring; and (3) fixes to the presentation issues you noted.

With about two days remaining in the discussion period, we would appreciate it if you could review our response and let us know whether it addresses your concerns.

Thank you again for your time.

Best regards, The authors of paper 46758

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.