acceptodds
Under review as a conference paper at ICLR 2027

EvoForge: The Dark Side of Evolving Memory

16 views

Abstract

As LLM agents take on increasingly complex tasks, memory becomes essential for accumulating experience and maintaining continuity. Moving beyond static record storage, modern memory mechanisms continually integrate information from past interactions, turning memory into an evolving system for future decisions. This evolution also introduces formation risk: attackers can exploit memory's own revision and integration mechanisms to turn scattered inputs into guidance that steers future agent behavior toward adversarial goals. In light of this, we propose EvoForge, an adaptive framework that derives a behavioral memory fingerprint from benign probes to guide attack construction and diagnosis. EvoForge submits complementary fragments through ordinary interactions to limit the adversarial intent exposed in each input, then uses behavioral feedback to revise their content and dependencies as memory evolves. Across three agent tasks and five memory configurations, EvoForge achieves an average attack success rate of 87.55, exceeding baselines in each setting by an average of 37.69. Further analyses characterize attack robustness and substantiate the contributions of EvoForge's core mechanisms.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agent, Agentic Memory, Memory Attack

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1BNwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The authors investigate a relatively interesting problem: memory attacks that are not injected all at once, but instead are broken into fragments and assembled through the memory management cycle. They propose a three-stage attack pipeline, starting from behavioral fingerprinting, then compositional fragment construction, and finally adaptive fragment evolution. Empirically, the paper reports an average ASR of 87.55, much higher than the chosen baselines, along with ablations that help show the robustness of the attack.

Critical strengths
  • The paper provides a comprehensive experiments across different memory backends, task domains, and baselines, along with ablations on the main stages of the attack.
Critical weaknesses
  • The paper is not well written. The main method, including the behavioral fingerprinting, fragment construction, and fragment evolution stages, remains too abstract even after I read the appendix. This is a critical weakness for a paper whose main contribution is methodological. In particular, it is still not clear to me how the fragments are actually submitted as ordinary interactions, or how the adaptive revision loop works in practice given how dataset-specific the setup appears to be. I had to rely heavily on the appendix to piece together the procedure, and even then some parts still feel under-specified. This significantly hurt my ability to understand and evaluate the paper.
  • The novelty also feels narrower than what the authors claim. While the paper emphasizes risks in evolving memory systems, the overall idea still reads to me as a combination of existing directions in memory poisoning, multi-fragment injection, and iterative refinement. Even though the empirical results look strong, the clarity issues make it hard for me to tell whether there is a deeper novelty here beyond packaging these existing ideas together in one framework.
Questions for authors
  • Can the authors provide one concrete running/end-to-end attack example, showing exactly how the behavioral fingerprinting, fragment construction, and fragment evolution stages work in practice, and how they ultimately lead to the final attack?
  • Can the authors clarify what is the novelty and how is fundamentally new compared to prior research?
Reviewer_iXxaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates formation risk in evolving memory systems for LLM agents, where memory updates can consolidate and connect scattered adversarial inputs into persistent knowledge that influences future agent behavior. The authors propose EvoForge, a black-box memory poisoning framework that uses benign probes to fingerprint memory behavior, constructs complementary attack fragments across four semantic roles, and iteratively refines them based on diagnostic feedback. The framework aims to exploit memory evolution through ordinary user interactions without directly modifying stored memory. Experiments across three agent tasks (financial reasoning, medical data queries, and shopping) and five memory mechanisms demonstrate an average attack success rate of 87.55%, outperforming four existing baselines. Additional ablation studies, sensitivity analyses, and defense evaluations assess the contributions of the proposed components and the attack's robustness across different settings.

Critical strengths

S1. Formalization of Formation Risk. The paper provides a clear formulation of memory evolution as a state transition process and defines formation risk in terms of its impact on downstream agent behavior. This formalization offers a useful perspective for understanding security risks arising from evolving memory systems.

S2. Systematic Attack Framework. EvoForge presents a well-structured attack framework combining behavioral memory fingerprinting, compositional fragment construction, and adaptive revision. These components address practical challenges in black-box memory poisoning, including uncertainty about memory mechanisms and limited visibility into memory states, making the framework applicable across different memory architectures.

Critical weaknesses

W1. The main concern is that the evaluation does not sufficiently isolate the security risks introduced by memory evolution. While the paper demonstrates strong attack success rates across several evolving memory systems, it remains unclear whether these successes arise from memory formation mechanisms, such as consolidation and linking, or from conventional memory poisoning through retrieval and downstream reasoning. Although Appendix F provides an illustrative example of memory updates, the paper lacks a controlled comparison with static memory under the same attack conditions. Such an experiment would be important to substantiate the central claim that memory evolution introduces additional attack risks.

W2. The defense evaluation also raises questions about the paper's claims regarding attack robustness. While the authors emphasize that EvoForge remains effective under various defenses, the results show substantial reductions in attack success rates, particularly under AMemGuard, where ASR drops from 81.11% to 10.00%.

W3. The novelty relative to existing compositional memory attacks could be clarified. In particular, the distinction between EvoForge's fingerprinting and adaptive revision mechanisms and those of prior approaches such as FragFuse and MemIncept is not entirely clear.

W4. Finally, the threat model describes shared-memory and memory-migration scenarios in which attacker-controlled interactions influence subsequent legitimate users. However, the experimental setup does not explicitly clarify how this separation is instantiated. I assume the evaluation follows the threat model, with attack interactions and subsequent legitimate queries treated as distinct sessions sharing the same persistent memory. It would be helpful for the authors to clarify this setup, including whether separate user identities or authorization boundaries are modeled, to establish how the experiments correspond to the stated threat model.

Questions for authors

Q1. Why does EvoForge remain vulnerable to defenses such as AMemGuard?

Q2. How does EvoForge fundamentally differ from FragFuse and MemIncept?

Q3. Are attacker and victim sessions separated as described in the threat model, and are distinct user identities modeled?

Reviewer_XyhvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper studies attacks that exploit how persistent memory changes over time. The paper’s focus on memory formation is relevant, and the reported results are striking. However, the evaluation leaves important questions about comparison fairness, uncertainty, and how well the tested setup represents the claimed attack scenarios.

Critical strengths

The paper considers how memory updates can affect later behavior, beyond whether an injected record is retrieved. It evaluates multiple memory configurations, task domains, attack baselines, and defenses. Ablations suggest that the framework’s components contribute unequally to attack success.

Critical weaknesses

EvoForge uses profiling, feedback, and revision, while some baselines are fixed attacks. The paper should compare methods under matched interaction and query budgets, or report cost alongside success.

Headline attack rates and baseline gaps lack confidence intervals or significance tests. Without repeated runs or paired uncertainty estimates, it is hard to tell how stable the reported advantage is.

The targets are predefined changes to answers or actions in controlled tasks. This demonstrates task redirection, but does not establish the same success rate for broader real-world harms.

The paper motivates shared-memory and memory-migration scenarios, but the experiments do not establish how often an attacker can write to memory that later affects a different user or deployment.

Defenses are evaluated on the Medical setting rather than across the full set of tasks and memory configurations, limiting the scope of the robustness claim.

Questions for authors

Can you report attack success with matched interaction/query budgets and include the total cost for each method?

Can you add repeated-run or paired uncertainty estimates for the headline comparisons, and test whether the advantage holds beyond the current fixed task setups?

Comments on the paper

1 comment
Area_Chair_HRYuOpenReview ↗

Reminder: Please Read and Discuss the Author Rebuttal

Dear Reviewers,

As the discussion period ends on November 18 (AoE), please take this final opportunity to review the authors’ rebuttal and discuss any remaining concerns with them. Even if the rebuttal does not change your assessment, I would appreciate it if you could briefly acknowledge that you have read and considered it.

Thank you for your time and contribution.

AC, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.