acceptodds
Withdrawn from ICLR 2027

Parallax: A Two-Phase Defense for Memory-Augmented LLM Agents from a Causal Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory poisoning allows adversaries to implant persistent records that steer LLM agents toward unauthorized actions across future queries. Content-level filters struggle with benign-looking, cooperative records, while action-level checks can block execution without removing the memory source of the threat. We propose PARALLAX, a defense that connects online interception with offline attribution through a causal perspective. Online, PARALLAX constructs counterfactual memories along the epistemic, procedural, and episodic pathways, aiming to suppress malicious influence while preserving beneficial task-relevant content. It compares actions under original and counterfactual memories in synchronized execution contexts, blocking actions that differ substantively and lack support from the user query, then prompting replanning. Offline, we define minimal trigger sets (MTSs) based on causal sufficiency and necessity for triggering an intercepted action. To accelerate the search for MTSs, PARALLAX generates candidate sets using an LLM guided by replay feedback and prior attribution experience, verifies them through repeated replay, and removes verified sets from memory. Across three agents and three adaptive memory-poisoning attacks, PARALLAX achieves the lowest attack success rate (0% on EHRAgent and StateAct) and 83.3–100.0% attribution recall at 100.0% precision, while preserving benign utility. These results demonstrate the advantages of a causal perspective for memory-poisoning defense.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Memory-Augmented LLM Agents, Memory Poisoning, Agent Security, Causal Defense

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WrTAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PARALLAX, a two-phase defense against memory poisoning in LLM agents from a causal perspective. The online phase uses counterfactual memory rewriting and dual-track execution to detect and block suspicious actions. The offline phase identifies and removes the responsible memory records through replay-based causal attribution. Experiments across multiple agents and attacks demonstrate strong defense effectiveness.

Critical strengths
  • Interesting and important problem. The paper addresses persistent memory poisoning, which is an important security concern for memory-augmented agents.

  • Interesting perspective. I appreciate the use of counterfactual reasoning to connect online action interception with offline memory attribution and cleanup.

  • Strong experimental results. The paper evaluates multiple agents, attacks, and defense baselines, showing strong improvements in attack prevention and attribution accuracy.

Critical weaknesses
  • (Main concern) Efficiency and scalability concerns. The online phase requires an additional counterfactual execution track at every action boundary, while offline attribution reportedly takes approximately 300 seconds per intercepted event. Moreover, attribution is evaluated with only four retrieved memory records. It remains unclear how the framework would scale to long-horizon agents with larger memory contexts. A comparison with a simpler intent-grounding-only guard would also help justify the additional computational cost.

  • Unclear reliability of counterfactual reasoning. The proposed method relies on rewriting memory and comparing the resulting actions. However, an action difference does not necessarily indicate malicious influence, since legitimate memories are also expected to affect agent behavior. It remains unclear how reliably the method distinguishes benign memory-dependent decisions from malicious ones, particularly in more complex tasks.

  • Causal attribution does not necessarily identify malicious records. The offline phase identifies minimal trigger sets through replay, but a record that contributes causally to a suspicious action is not necessarily attacker-poisoned. Removing entire trigger sets could therefore delete useful benign memories. I would appreciate further analysis of this distinction and its impact on memory utility.

  • Limited proactive coverage and long-term evaluation. Offline attribution is triggered only after an action has been intercepted, leaving poisoned memories that have not yet caused detectable suspicious behavior unexamined. Furthermore, the paper does not sufficiently evaluate whether memory cleanup prevents future attacks while preserving long-term agent utility across subsequent sessions.

Questions for authors

See weaknesses.

Reviewer_56SEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PARALLAX memory-poisoning defense for LLM agents that combines online and offline actions. The method includes rewriting the memory (in a counterfactual manner) to determine whether retrieved memories are causing an agent to take unauthorized actions. When a suspicious action is detected, it is blocked, and an offline attribution procedure identifies the memory records responsible through replay and minimal trigger set identification. The paper evaluates the method on 3 attacks and 3 agents.

Critical strengths
  • There are many fewer defense strategies for memory poisoning; hence this work's thesis is quite relevant.

  • Their methodology of online interception with offline remediation is interesting, and can inspire some future works.

  • The method shows strong performance across benchmarks.

Critical weaknesses
  • The paper repeatedly claims to evaluate against "adaptive attacks" (also in the abstract). However, their definition (attack-strategy adaptivity) of adaptive is probably very different from an adaptive adversary in security, where the attacker has knowledge of the defense. I only learned about the new definition in sec 5.1.

  • Online tool detection focuses on unauthorized tool actions. Memory attacks can be broader, such as belief manipulation (poisioning false memory about their political stance or that user prefers a certain prodcut)

  • The experiments don't have true adaptive and stronger memory poisoning attacks such as [1], where they have they evaluate addaptive attacks.

  • Some of the variables make it harder to read the paper, for example variables such as B' etc are never reused later.

[1] Hidden in Memory: Sleeper Memory Poisoning in LLM Agents

Questions for authors
  • Because PARALLAX's counterfactual operators weaken assertions and directives, legit memory-dependent behavior may also be suppressed?! I am not sure if exp in 5.4 solves this.

  • How does PARALLAX's defense work when evaluated on an adaptive attack like in [1]?

  • the authors aim to maximize retained benign memories, but the paper reports attribution precision and recall primarily for poisoned records. This doesnt measure or characterize the costs of removing legit memories or the effect of pruning on future user tasks.

Reviewer_GbjXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PARALLAX, a two-phase defense against memory poisoning attacks on LLM agents. In the online phase, PARALLAX constructs counterfactual memories by weakening factual assertions, converting instructions into historical descriptions, and masking task-specific actions. It then compares the agent's actions under the original and counterfactual memories, blocking actions that differ substantially and are not supported by the user's request. In the offline phase, the method uses LLM-guided proposals and repeated agent replays to identify minimal sets of memory records responsible for suspicious actions and removes them from persistent memory. Experiments across three agents and three memory poisoning attacks show substantially reduced attack success rates and high attribution accuracy.

Critical strengths

The paper addresses an important security problem in memory-augmented LLM agents, particularly cooperative memory poisoning attacks where multiple benign-looking records jointly induce malicious behavior. The combination of online interception and offline attribution is interesting because it not only prevents suspicious actions but also attempts to eliminate their underlying causes from persistent memory. The minimal trigger set formulation is especially useful for identifying redundant or overlapping groups of poisoned records that conventional leave-one-out attribution may miss. The evaluation is relatively comprehensive, covering multiple agents, attack methods, defense baselines, and ablation studies, with strong reported results.

Critical weaknesses

My main concern is the practicality and scalability of the proposed approach. PARALLAX requires counterfactual memories to be generated for each memory record and dual-track LLM inference at every action boundary, even when no attack is present. Although counterfactual memories can be cached, generating and maintaining transformed copies could introduce substantial preprocessing and storage overhead, especially in real-world memory systems containing millions of records. More importantly, caching does not eliminate the need for an additional LLM inference at every action boundary, potentially doubling the action-generation inference cost for long-horizon agents. The offline attribution phase further increases computational costs through repeated agent replays. These limitations raise serious concerns about whether the approach can scale to production-level memory-augmented agents. Additionally, counterfactual rewriting may alter legitimate memory information rather than selectively removing malicious influence. As a result, differences between the original and counterfactual actions do not necessarily indicate an attack, making the approach heavily dependent on the reliability of an additional LLM judge. The paper would benefit from a scalability analysis involving substantially larger memory banks and realistic long-horizon agent workloads.

Questions for authors
  1. what is the cost/latency of running PARALLAX? I am highly concerned with practical aspect of this approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.