acceptodds
Under review as a conference paper at ICLR 2027

OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences

Abstract

Memory-augmented LLM agents improve across tasks by reflecting on past trajectories and distilling reusable rules. This creates a distinct attack surface: an experience may be correct in its source context yet unsafe when generalized beyond it. Existing agent-memory attacks often rely on malicious payloads, triggers, or explicit unsafe patterns that content-level defenses can target. We introduce OEP, an input-only black-box attack that targets the generalization step of self-evolution rather than the memory store. OEP constructs boundary cases where a non-standard method is locally valid but poorly transferable, then reinforces that lesson with severe yet semantically plausible consequence framing. Using only normal user-level interaction, OEP can bias reflection toward persistent over-generalized rules that degrade later benign tasks. In our main GPT-4o setting, OEP reaches ASRs of 59.1%, 52.0%, and 71.9% on math, medical reasoning, and tool use. Under an LLM auditor on GSM8K, OEP retains 40.3% ASR while the evaluated baselines fall below 15%. These results expose an experience-transferability gap: episode-level validity does not guarantee the safety of the reusable rule distilled from it. Our work is available at https://anonymous.4open.science/r/OEP-Attack-D3D2/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, agent, memory, attack

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bJA7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OEP, an attack against agent memory that leverages edge-case experiences to mislead the agent's self-evolution. The attack is designed by injecting locally valid but non-transferable experiences into the agent's memory, considering that the injection should bypass a memory filter. The payload is designed as an adversarial triplet that articulates the consequences if the injected target method is not taken. The proposed attack has been tested on three datasets.

Critical strengths
  • The paper studies a timely problem. Compared to prior work that identifies the "misaligned experience reply" phenomenon that hinders agent self-evolution from memory, this work proposes a method to amplify the "misalignment".
Critical weaknesses
  • Over-strong assumption: Agents typically execute a task and then store its abstracted execution as an experience in memory. The user only submits the task query. However, in this work, the user submits the adversarial experiences.

  • The method is non-principled, as the edge-case construction was left to an LLM without explicit formulation of distribution shift.

  • Novelty: The proposed method merely amplifies task distribution shift. Simple methods such as confidence checks may easily defend against the attack by prompting additional inspection when the past experience is deemed not applicable.

  • Experiments: The proposed method was tested on simple tasks (Math, MedQA, and Tooluse) with simple agentic paradigms, not comparable to prior work testing on more complex tasks (Autonomous driving, MIMIC, and WebShop).

  • The proposed attack was not tested against prompt injection detection.

Minor issues

  • Line 248: Please capitalize 's' in "safety-aligned".
  • Xiong et al. (2026) and Xiong et al. (2025b) refer to the same paper
Questions for authors
  • The proposed method is claimed to be compared with Zhan et al. (2024), Dong et al. (2025), and Chen et al. (2024). But I couldn't find the corresponding results.
Reviewer_yNuMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important problem in self-evolving LLM agents: the misgeneralization of locally valid but non-transferable experiences, which can lead to unexpected and harmful downstream behaviors. The authors propose Obsessive Experience Poisoning (OEP), an attack that constructs carefully selected corner cases and amplifies their consequences to induce agents to learn and retain over-generalized rules through reflection. Experiments across multiple domains, models, and agent frameworks demonstrate the effectiveness and persistence of the attack, highlighting the security risks associated with experience-based self-evolution.

Critical strengths
  • Interesting and important problem. The paper investigates an important vulnerability in self-evolving agents, where locally valid experiences can be incorrectly generalized into harmful rules. I find this problem interesting and relevant to the security of memory-based agents.

  • Clear writing and motivation. The paper is generally well written, and the motivation and overall attack pipeline are clearly presented.

  • Promising experimental results. The experiments demonstrate the effectiveness of OEP across multiple settings, including its persistence and robustness against the evaluated defenses.

Critical weaknesses

Weaknesses

  • Limited systematic and automated attack construction. The corner-case construction process relies heavily on LLM generation and human-guided refinement. Although the authors define local validity and non-transferability as formal constraints, it remains unclear how non-standard methods are systematically discovered and how much human intervention is required. Moreover, the choice of 10 injected experiences is insufficiently justified. An evaluation under different experience budgets would help establish the generality and practicality of OEP.

  • Limited exploration of the reinforcement mechanism. The reinforcement stage mainly relies on one factor that is amplifying the consequences of corner-case experiences. While the ablation studies demonstrate that stronger consequence framing improves attack success, it remains unclear whether other factors, such as experience relevance, repetition, or source credibility, play equally important roles. A more systematic analysis would strengthen the methodological contribution.

  • (minor but important) Limited evaluation of high-impact security consequences. Although the paper covers math, medical reasoning, and tool use, the evaluation primarily demonstrates incorrect answers and increased resource consumption. It would be valuable to extend the evaluation to realistic agent workflows involving concrete security consequences, such as unauthorized actions, data exfiltration, or financial loss in controlled environments, to better establish the practical severity of the vulnerability.

  • (minor) Insufficient isolation and understanding of the underlying mechanism. Although the paper provides several useful ablations, important questions remain about how poisoned experiences are retained, retrieved, and applied in downstream tasks. In particular, direct-case retrieval already achieves substantial attack success without explicit reflection, raising the question of how much of OEP's effectiveness is specifically attributable to reflection-induced over-generalization rather than ordinary experience-following. A more fine-grained analysis of these stages would provide valuable insights beyond demonstrating attack effectiveness.

While I find the problem of harmful experience over-generalization interesting and important, I am less convinced by the depth of the technical contribution. The method largely combines heuristic corner-case construction with consequence framing, while several fundamental questions about the underlying mechanism remain insufficiently explored. Some questions can be: 1) could the corner case discovery be discoverd without human intervention? 2) Is there other factors other than "consequence" framing can be used? How to find a comprehensive set of such cases? It raises an interesting scientific question but does not investigate it deeply enough. In particular, the paper does not clearly isolate the contribution of reflection-induced generalization from ordinary experience-following, nor systematically characterize when poisoned experiences are retained, retrieved, and applied in downstream tasks. I believe a deeper understanding of these mechanisms would substantially strengthen the paper's contribution.

Questions for authors

See weaknesses.

Reviewer_2WW7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Obsessive Experience Poisoning (OEP), an attack targeting self-evolving LLM agents that learn from previous interactions and consolidate their experiences into persistent memory. The main idea is to exploit the tendency of these agents to generalize from past experiences by introducing examples that are valid in their original context but inappropriate when applied to other tasks. The proposed framework consists of three stages: constructing clean edge cases, using Adversarial Consequence Triplets (ACT) to influence the agent's preferences, and encouraging the agent to incorporate these experiences into reusable operational rules. The authors evaluate OEP across mathematical reasoning, medical reasoning, and tool-use tasks using several LLM backbones and self-evolving agent frameworks. The experiments report high attack success rates, demonstrate persistence across subsequent interactions, and compare the proposed approach against existing attacks and defenses. The paper also includes ablation studies and an analysis of the mechanisms that may explain the observed behavior.

Critical strengths

The paper addresses an interesting security problem that deserves further attention. As LLM agents increasingly rely on accumulated experiences to improve their performance, understanding how these experiences can be manipulated is important. I particularly appreciate the authors' focus on the distinction between an experience being correct in a specific situation and the same experience being unsuitable as a general rule. This is a meaningful observation that could have implications for the design of more secure memory mechanisms. The proposed attack is also reasonably well structured. The combination of edge-case construction, consequence manipulation, and memory consolidation provides an understandable account of how the attacker attempts to influence future agent behavior. The paper is generally easy to follow, and the examples help illustrate the intended attack mechanism. I also appreciate the experimental effort. The authors evaluate multiple domains, models, and agent frameworks, which provides a broader picture of the attack's behavior than a single-model evaluation would. The persistence experiments, ablation studies, and investigation of potential defenses are useful additions.

Critical weaknesses

My main concern is the novelty of the proposed attack relative to existing memory poisoning methods. While the authors emphasize that OEP relies on locally valid but non-transferable experiences rather than explicitly malicious instructions, I am not convinced that this distinction is sufficiently established experimentally. At a broader level, the attack still involves manipulating the information an agent learns from and causing it to adopt inappropriate rules for future decisions. The paper would benefit from a more convincing demonstration of what makes this mechanism fundamentally different from existing approaches. The current experiments suggest that the attack can be effective, but they do not fully isolate the contribution of the proposed non-transferability property. I am also concerned about how the authors establish that the injected experiences are genuinely valid in their original contexts. This is a central assumption of the paper, yet the validation procedure is not sufficiently convincing. In particular, the medical reasoning experiments involve situations where contextual correctness can be difficult to determine without expert judgment. If the injected examples are not consistently valid, then the attack becomes much closer to conventional poisoning with misleading information. I would have liked to see stronger independent verification of both local validity and non-transferability. Another issue is the threat model. The paper describes OEP as an input-only black-box attack, but it is not entirely clear how much information the attacker needs about the target agent's memory and reflection mechanisms. The construction of adversarial experiences appears closely related to the way the target system consolidates information. This raises questions about whether the attack would remain effective against agents with substantially different or unknown memory architectures. The authors should clarify the attacker's knowledge and capabilities more carefully and provide stronger evidence that the method is practical under the stated assumptions. I also find the experimental comparisons somewhat difficult to assess. Although the reported improvements over existing attacks are promising, the paper does not sufficiently establish whether the methods are compared under equivalent conditions. Differences in attack budgets, optimization procedures, and compatibility with the evaluated agent frameworks could influence the results. Since OEP is specifically designed for experience-based memory consolidation, comparing it against attacks with different objectives may not provide a fully fair assessment of its advantages. More carefully controlled baselines are needed before concluding that OEP is consistently superior. The statistical reliability of the results is another concern. The paper reports many numerical improvements, but the main tables do not provide enough information about variability across independent runs. Given the stochastic behavior of LLM agents, this makes it difficult to judge whether some of the reported differences are robust. Reporting confidence intervals or standard deviations across independent attack constructions and agent executions would make the empirical conclusions more convincing. Finally, I am not fully persuaded by the mechanistic analysis. The mathematical formulations involving rule adoption, consequence severity, and memory consolidation provide an intuitive explanation of why the attack might work. However, the connection between these formulations and the actual behavior of the evaluated agents is not demonstrated rigorously. As a result, the analysis feels more like an interpretation of the experimental observations than an independently validated explanation. This does not necessarily undermine the empirical results, but it limits the strength of the theoretical contribution.

Questions for authors
  1. What evidence demonstrates that the non-transferability of locally valid experiences is the main factor responsible for OEP's effectiveness, rather than general memory contamination?
  2. How do the authors independently verify the local validity of adversarial experiences, especially in medical reasoning tasks?
  3. How much knowledge of the target agent's memory consolidation mechanism is required to construct a successful attack? Would OEP remain effective if this mechanism were completely unknown?
  4. Can the authors provide experiments that distinguish persistent memory poisoning from temporary contextual influence, for example by disabling memory consolidation while keeping the remaining experimental conditions unchanged?
  5. Were the baseline attacks evaluated with equivalent attack budgets and optimization opportunities? How many independent runs were performed, and how much variability was observed? Final Assessment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.