acceptodds
Under review as a conference paper at ICLR 2027

From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents

12 views

Abstract

As LLM agents increasingly rely on persistent memory for long-horizon and personalized behavior, they can retain and reuse information across interactions, but this also creates a lasting channel through which malicious memory writes can influence future behavior. Persistent-memory attacks are typically evaluated by whether they succeed, yet successful attacks can leave persistent states with substantially different downstream consequences. We study this severity as a distinct attack-design objective and formalize it with **counterfactual memory regret (CMR)**, the paired increase in expected downstream loss relative to clean memory. We introduce **MemHarm**, which predeclares a finite class of sparse, grounded semantic edits, evaluates candidates through the normal agent memory interface using offline paired-loss feedback, and certifies resolved selections within that class. Compared with attack-success optimization, CMR-guided selection produces substantially larger downstream loss while retaining most of the success-rate gain. Across two agent benchmarks and diverse memory designs, MemHarm attains the highest CMR point estimates among the evaluated general attacks on identical support. Factor-removal interventions link this harm to the selected semantic factor, and native-agent deployments verify the write-to-fresh-process attack path.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Memory, Agentic AI, Memory Attacks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YToqOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a more fine-grained fitness function for optimizing memory poisoning attacks. Instead of relying on binary attack success, it proposes Counterfactual Memory Regret (CMR), which measures the difference in downstream operational loss between clean and poisoned memory. I find the counterfactual formulation particularly interesting, as it provides a meaningful baseline for quantifying the actual impact of an attack. The paper also provides extensive experimental evaluations to demonstrate the effectiveness of the proposed approach.

Critical strengths
  • Clear writing and motivation. The paper is well written, and the motivation for moving beyond binary attack success toward severity-aware optimization is clearly presented.

  • Comprehensive experiments. The evaluation is thorough, covering multiple benchmarks, memory architectures, and representative baselines. The results demonstrate that MemHarm consistently achieves higher attack severity than existing methods, including strong baselines.

  • Interesting mechanism findings. I particularly appreciate the finding that the specific corrupted semantic factor matters. This aligns with my intuition that the effectiveness of prompt injection often depends on how relevant the injected information is to the agent's task. It is interesting to see the paper provide empirical evidence of the causal contribution of individual memory factors to downstream attack severity.

Critical weaknesses
  • Evaluation on relatively outdated models. The main CMR experiments primarily use Llama-3.1-70B and GPT-4o, which are relatively outdated compared with recent models. Although the paper includes additional evaluations with Gemini-3.5-Flash and GPT-OSS-120B, it remains unclear whether the observed improvements would generalize to more capable, recent agents. This is very important as it can provide evidence to show whether the CMR algorithm can still be interesting if the models get stroner. Evaluating stronger models under the same CMR protocol would make the findings more convincing.

  • Overstated theoretical contributions. While Proposition 1 provides useful justification for the optimization procedure, the severity decomposition and subsequent bounds are relatively straightforward derivations, and Proposition 2 follows from standard statistical arguments. More importantly, these results do not appear to directly inform the algorithm design or provide substantial new insights into its effectiveness. I would encourage the authors to clarify their practical implications or simplify the theoretical presentation.

Questions for authors

While I think the paper provide very interesting finding and solid experiment results, my main concern is that the evaluation is still based on some outdated models which make it still unclear whether it can be effective for stronger LLM.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YToq

We thank the reviewer for emphasizing stronger-model generalization and the practical significance of the theory. We conducted a new matched-protocol evaluation on a contemporary backbone and will sharpen the presentation of the theoretical results.

Response to W1 and Q1. Stronger-Model Generalization

Appendix G.1 already reports CMR results on Gemini-3.5-Flash and native-agent evaluations on GPT-OSS-120B. To further examine generalization under the full paired-CMR protocol, we evaluated GPT-6.1 Terra on AgentDojo and τ³-bench, with 48 held-out tasks per benchmark and matched memory contracts, evaluation seeds, and attack budgets (Table Y1).

Table Y1. Stronger-backbone CMR evaluation on GPT-6.1 Terra (new)

BenchmarkASR-guided CMRBest matched baseline CMRMemHarm CMRMemHarm − ASR (95% CI)
AgentDojo0.3010.0340.418+0.117 [+0.035, +0.213]
τ³-bench0.2160.1910.274+0.058 [-0.027, +0.143]

To contextualize these attack results, Table Y2 compares the stronger defender's clean-task utility with the original GPT-4o reference.

Table Y2. Clean-task utility: original and new defender evaluations

DefenderAgentDojo clean utility (%)τ³-bench clean utility (%)
GPT-4o (original reference)73.863.1
GPT-6.1 Terra (new evaluation)81.074.0

On GPT-6.1 Terra, MemHarm achieves the highest CMR on both benchmarks, outperforming ASR-guided selection by +0.117 on AgentDojo (paired 95% CI [+0.035, +0.213]), with a positive gain also observed on τ³-bench. The defender maintains strong clean-task utility across both benchmarks. These findings extend the evidence for MemHarm's effectiveness to a more capable tool-using backbone under the original paired-CMR evaluation protocol.

Response to W2. Theoretical Contributions

We appreciate the reviewer's suggestion to clarify the practical role of our theoretical analysis.

Proposition 1 directly informs attack selection: its confidence bounds guide rollout allocation and provide a stopping criterion for certifying an η-near-optimal semantic edit. Appendix E reports 91.4% certified instances and 30.7% fewer median paired rollouts at the same certification rate.

The severity decomposition provides a mechanistic interpretation by separating the downstream consequences of corrupting a semantic factor from the memory retrieval and use processes that realize those consequences. Consistent with this distinction, Appendix G.3 shows a 0.34 CMR decrease after removing the targeted factor, compared with 0.02 for an irrelevant factor.

We will clarify these algorithmic and empirical connections while moving Proposition 2 to the appendix to streamline the theoretical presentation.

Summary and Planned Revisions

The stronger-backbone experiment further demonstrates MemHarm's generalization to more capable defenders, while the existing certification and factor-removal results clarify the theory's practical contribution.We will add the new results, report their uncertainty, and streamline the theoretical presentation around its algorithmic and mechanistic roles.

We are especially grateful for the reviewer's insightful suggestions, which have guided us toward a more rigorous evaluation and a clearer articulation of the theory's practical value.

Thank you for your response and the additional experiments. Regarding stronger models, I meant models with greater capabilities, rather than simply newer models—for example, gpt-soll rather than only terra. I also appreciate the clarification of the theoretical contribution. Please incorporate these clarifications into the revised manuscript.

Thanks for your feedback. We will run the suggested experiment and incorporate the results and clarifications into the revision.

Reviewer_GfcAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the severity of persistent-memory attacks against LLM agents, arguing that binary attack success does not adequately capture downstream consequences. It introduces Counterfactual Memory Regret (CMR), defined as the expected increase in normalized operational loss between edited and clean memory states. Based on this objective, the authors propose MemHarm, which constructs a finite set of sparse, grounded semantic memory edits and uses adaptive paired-loss evaluation to select high-CMR candidates with class-conditional statistical guarantees. The paper evaluates MemHarm on AgentDojo and τ³-bench across different models and memory architectures, comparing it with existing memory-poisoning attacks and conducting ablation and sensitivity analyses. Additional experiments on OpenClaw and Hermes using MPBench evaluate whether the attacks can be written into persistent memory and reactivated in fresh agent processes. The results show that MemHarm achieves higher CMR than the evaluated baselines while maintaining competitive attack-success metrics.

Critical strengths

S1. Careful experimental design and extensive validation. The paper provides a systematic evaluation across multiple benchmarks, model backbones, and memory architectures. They help isolate the contributions of the proposed method. The native-agent experiments further demonstrate that the attack can operate through real persistent-memory interface.

S2. A well-defined optimization and statistical evaluation framework. The paper provides a clear mathematical formulation of CMR and develops a structured procedure for constructing and selecting memory edits. The use of paired evaluations and class-conditional statistical guarantees makes the optimization procedure more rigorous and interpretable.

Critical weaknesses

W1. Unclear significance of the proposed objective. CMR measures normalized defender-task utility degradation, but its relationship to security-relevant attack severity is not sufficiently built. Higher task loss does not necessarily implies more consequential attacker-induced outcomes.

W2. Limited validation of CMR as a severity metric. The paper demonstrates that MemHarm can effectively optimize CMR, but provides limited independent evidence that higher CMR corresponds to more severe security consequences. The operational-priority sensitivity analysis does not fully address this concern.

W3. Nonstandard ASR definition. The main experiments define ASR as task regression rather than explicit attacker-goal achievement (as in AgentDojo). Consequently, the ASR-guided versus CMR-guided comparison mainly contrasts binary and continuous utility-task degradation, rather than conventional attack success and attack severity.

W4. Unclear benefits over existing poisoning attacks. Although MemHarm achieves higher CMR than existing methods under carefully controlled comparisons, it remains unclear whether this translates into higher attacker-goal success rates or more consequential security outcomes under the baselines' original attack objectives.

Questions for authors

Q1. Security relevance of CMR. Can the authors provide systematic evidence that higher-CMR attacks lead to more consequential security violations, rather than merely greater degradation of benign task performance? For instance, a breakdown of what kinds of failures underlie the reported CMR, or a comparison with a severity measure that reflects the magnitude of harm, would help.

Q2. Benefits beyond optimizing CMR itself. When evaluated using explicit attacker objectives or the native success criteria of existing poisoning benchmarks (e.g., AgentDojo’s security metric, i.e., injection-task success), does CMR-guided optimization offer any measurable advantages over conventional attack methods or attacker-goal-based ASR-guided selection?

8 comments
AuthorsOpenReview ↗

Response to Reviewer GfcA (1/3)

We appreciate the reviewer's focus on whether CMR offers security value beyond optimizing its own benchmark metric. We address this through independent security outcomes and a new evaluation using AgentDojo's native attacker-goal success criteria.

Response to W1. Significance of CMR

In this response, we clarify the role of CMR (1), show that it captures information beyond binary ASR (2), and provide independent evidence linking CMR-guided optimization to consequential security outcomes (3).

  1. Counterfactual severity objective. CMR measures the attack-attributable increase in downstream loss relative to clean memory, distinguishing attack-induced harm from pre-existing task failures. Unlike binary ASR, it captures how much downstream loss an attack causes, rather than only whether a task is compromised. Eq. (4) accommodates consequence-sensitive losses, while Eq. (2) adopts task-requirement loss for reproducible evaluation.

  2. Distinct information from binary attack success (Appendix H.2). Appendix H.2 demonstrates that the two objectives can favor different attacks:

  • ASR-guided: 75.30% ASR, 0.6958 CMR.
  • CMR-guided: 71.28% ASR, 0.7830 CMR.

Thus, a higher frequency of task compromise does not necessarily imply greater downstream loss. CMR captures a complementary dimension of attack effectiveness.

  1. Independent security relevance (full results in W2 and Q2).
  • Security-consequence audit (W2): With attacks frozen before independent evaluation, CMR-guided selection produces +12.4 pp unauthorized external actions (paired 95% CI [+2.0, +23.9]) and +0.108 consequence-sensitive regret (95% CI [+0.049, +0.181]) over ASR-guided selection.
  • Native attacker-goal evaluation (Q2): CMR-guided selection also yields a +8.9 pp higher independently measured severe-outcome rate (paired 95% CI [+1.8, +15.9]) than native goal-guided selection, despite a lower attacker-goal success point estimate.

These complementary evaluations link severity-guided optimization to observable security violations and independently assessed consequences, rather than improvements in the original task-loss objective alone.

We will make this conceptual and empirical connection more explicit in the revised paper.

Response to W2. Independent Severity Validation

  • New experiment: Security consequences beyond task loss. We evaluated frozen ASR-guided and CMR-guided attacks on 261 held-out task clusters using independent authorization rules, execution logs, and consequence-sensitive losses, with the results summarized in Table G1.

Table G1. Independent security-outcome audit (new)

Security outcomeASR-guidedCMR-guidedPaired difference (95% CI)
Unauthorized external actions (%)34.747.1+12.4 pp [+2.0, +23.9]
Policy-violating state changes (%)36.246.1+9.9 pp [-1.9, +21.7]
Mean unauthorized amount per eligible task327506+179 [-26, +384]
Consequence-sensitive regret0.2610.369+0.108 [+0.049, +0.181]

CMR-guided selection produces more unauthorized external actions and higher independently measured consequence-sensitive regret. This supports security consequences beyond benign task regression.

AuthorsOpenReview ↗

Response to Reviewer GfcA (2/3)

Response to W3. ASR Definition and Comparison

Thanks for this valuable question. We clarify three aspects of our ASR definition: (1) its distinction from native attacker-goal success, (2) its motivation in persistent-memory attacks, and (3) empirical validation under both criteria.

  1. Distinct evaluation questions. AgentDojo's native metric [1] asks whether an attack achieves a predefined attacker-specific goal, such as an unauthorized action. Our ASR (Section 5.1) instead asks whether memory poisoning causes a previously successful downstream task to fail, directly measuring attack-induced task compromise.

  2. Motivation for our ASR definition. A poisoned memory may compromise multiple future workflows without satisfying a single predefined attacker-goal predicate. Comparing clean and poisoned executions excludes pre-existing failures and provides a consistent binary measure across heterogeneous tasks. ASR therefore captures the frequency of attack-induced compromise, complementing CMR's measure of downstream loss magnitude.

  3. Empirical validation. Appendix H.2 shows that ASR-guided selection achieves 75.30% ASR / 0.6958 CMR, compared with 71.28% ASR / 0.7830 CMR under CMR guidance. In our new evaluation under AgentDojo's native attacker-goal criteria (Q2), CMR-guided selection additionally achieves a 8.9 pp higher independent severe-outcome rate (paired 95% CI [+1.8, +15.9]).

We will clarify the distinction between task-compromise ASR and conventional attacker-goal ASR throughout the paper, making their respective evaluation objectives and security interpretations explicit.

Response to W4. Benefits Over Existing Poisoning Attacks

  • New evaluation: Native attacker goals and security consequences. To directly address this concern, we evaluated MemHarm against three representative memory-poisoning attacks, AgentPoison [2], InjecMEM [3], and MemoryGraft [4], using matched AgentDojo-native attacker-goal criteria, CMR, and independently assessed severe outcomes (Table G2).

    Table G2. New comparison with named poisoning baselines under native attacker-goal evaluation

    MethodNative goal success (%) ↑CMR ↑Severe-outcome rate (%) ↑
    AgentPoison [2]50.90.01839.8
    InjecMEM [3]54.20.04742.6
    MemoryGraft[4]56.70.27745.8
    MemHarm62.50.44459.7

MemHarm achieves the highest native attacker-goal success (62.5%) and independently measured severe-outcome rate (59.7%) among the evaluated poisoning baselines. The separate security audit in W2 further supports this finding through increased unauthorized external actions and consequence-sensitive regret under CMR guidance.

These results demonstrate that MemHarm's benefits extend beyond its optimized CMR objective. We will include the named-baseline comparisons and clarify their evaluation criteria in the revision.

Response to Q1. Security Relevance of CMR

We evaluated frozen attacks using independent criteria covering unauthorized actions, policy violations, monetary consequences, and consequence-sensitive regret, none of which entered CMR optimization. As shown in Table G1 (W2), CMR-guided selection increases unauthorized external actions by 12.4 pp (paired 95% CI [+2.0, +23.9]), alongside higher consequence-sensitive regret.

We further evaluated attacks under AgentDojo's native attacker-goal criteria (Table G3, Q2). Compared with native goal-guided selection, CMR guidance achieves a 8.9 pp higher severe-outcome rate (paired 95% CI [+1.8, +15.9]). Together, these evaluations link CMR-guided optimization to independently measured security violations and downstream consequences.

These complementary experiments provide direct evidence that CMR-guided optimization translates into observable security violations and increased consequence-sensitive harm, rather than merely greater degradation of benign task performance.

AuthorsOpenReview ↗

Response to Reviewer GfcA (2/3)

Response to Q2. Native Attacker-Goal Evaluation

We thank the reviewer for raising this particularly useful question.

  • New experiment: Native attacker goals versus severity. To directly address this concern, we evaluated 120 matched AgentDojo task–goal clusters using the original injection-task success criteria. We compared goal-guided, CMR-guided, and goal-first/CMR-tie-break selection under the same semantic-edit space. Goal-first prioritizes attacker-goal success and uses CMR to select among near-optimal candidates; Table G3 compares these strategies alongside an existing-attack baseline.

    Table G3. New native attacker-goal and severity comparison

    Selection strategyNative goal success (%) ↑CMR ↑Severe-outcome rate (%) ↑
    Native goal-guided65.80.32950.8
    CMR-guided MemHarm62.50.44459.7
    Goal-first / CMR-tie-break65.30.41657.2
    Development-selected existing attack56.70.27745.8
    • Key findings.
      • CMR-guided selection: Achieves a +8.9 pp severe-outcome advantage (paired 95% CI [+1.8, +15.9]) over native goal-guidance, illustrating the additional security consequences captured by severity optimization.
      • Goal-first/CMR-tie-break: Maintains similar native-goal success point estimates while achieving a +6.4 pp higher severe-outcome rate (95% CI [-1.4, +13.6]), demonstrating the potential value of incorporating severity into goal-oriented selection.

Native attacker-goal success and downstream severity capture complementary dimensions of attack effectiveness. The results demonstrate the value of CMR-guided selection for identifying more consequential attacks, while goal-first/CMR-tie-break provides a practical way to combine both objectives.

Summary and Planned Revisions

The new experiments establish MemHarm's relevance under both independent security-consequence measures and AgentDojo's native attacker-goal criteria. We will clarify the two ASR conventions, strengthen the security interpretation of CMR, and incorporate the matched native-goal results and their uncertainty into the revised paper.

We sincerely appreciate the reviewer's constructive feedback, which has helped us strengthen both the empirical validation and the presentation of our core contribution.

References

[1] Debenedetti et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024.

[2] Chen et al. (2024). AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. NeurIPS 2024.

[3] Tian et al. (2026). InjecMEM: Memory Injection Attack on LLM Agent Memory Systems. arXiv:2608.23471.

[4] Srivastava et al. MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval. arXiv:2512.16962.

Reviewer_WAXFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the impact of the persistent memory attacks on the subsequent task performance of LLM agents. It introduces CMR as a metric to quantify the expected incremental loss caused by editing memory compared to clean memory. To achieve this object, the proposed method, MemHarm, constructs and freezes a finite set of semantic editing candidates. It then employs paired feedback and adpative sampling to select candidates that yied high loss, while providing a guarantee of near-optimality within the candidate set. The authors evaluate the approach across two benchmarks, multiple models, and six distinct memory mechanisms, and further demonstrate that these attacks can persist across different execution processes.

Critical strengths
  • The paper addresses a meaningful evaluation problem.
  • The evaluation includes thoughtful controls.
  • The experiments cover several complementary aspects of the attack.
Critical weaknesses
  • The connection between CMR and operational severity needs stronger justification. Equation (2) primarily measures the average proportion of unmet task requirements. With a single binary requirement, CMR reduces to the difference in failure probabilities between the attacked and clean conditions. With multiple requirements, it captures additional requirement violations, but does not necessarily distinguish their practical consequences. For example, sending an email to the wrong recipient may have substantially different implications from omitting an attachment, although both can contribute equally to the loss. The reweighting analysis in Appendix G.2 is helpful, but does not directly establish that CMR captures the monetary magnitude, irreversibility, or cumulative consequences motivating Figure 1. The paper should more clearly delimit its notion of severity or provide a targeted evaluation using consequence-sensitive losses.
  • The attack assumes substantial knowledge of downstream tasks. CMR depends on the downstream task distribution and operational loss, yet an attacker may not know which tasks an agent will perform after memory poisoning. MemHarm assumes distribution-matched optimization tasks and access to paired-loss feedback. While this supports controlled evaluation, it remains unclear how an attacker obtains such information in practice or whether the selected edits remain harmful on unseen task types. The paper should better justify this assumption and evaluate transfer under task-distribution mismatch.
Questions for authors

See the weaknesses part

1 comment
AuthorsOpenReview ↗

Response to Reviewer WAXF

We thank the reviewer for highlighting two important questions: whether CMR reflects meaningful security consequences and whether attacks transfer beyond the optimization task distribution. We conducted targeted experiments addressing both.

Response to W1. Operational Severity

CMR measures attack-attributable downstream loss under predefined evaluation criteria. Its general formulation (Eq. 4) accommodates consequence-sensitive losses, while Eq. (2) provides a reproducible task-requirement instantiation.

To directly examine the operational significance of this formulation, we evaluated frozen ASR-guided and CMR-guided attacks on 261 held-out tasks using independent security criteria excluded from optimization. Both methods used matched attack spaces and paired evaluation, with task-cluster bootstrap confidence intervals reported in Table W1.

Table W1. Independent security-consequence evaluation (new)

Security outcomeEligible tasksASR-guidedCMR-guidedPaired difference (95% CI)
Unauthorized external actions (%)19034.747.1+12.4 pp [+2.0, +23.9]
Policy-violating state changes (%)17636.246.1+9.9 pp [-1.9, +21.7]
Mean unauthorized amount per eligible task133327506+179 [-26, +384]
Consequence-sensitive regret2610.2610.369+0.108 [+0.049, +0.181]

CMR guidance increases independently verified unauthorized external actions and consequence-sensitive regret, with positive confidence intervals for both differences. Policy violations and monetary magnitude are directionally positive. This directly supports CMR's operational security relevance beyond task incompleteness.

Response to W2. Attacker Knowledge and Distribution Shift

CMR-guided selection uses available source-task samples and paired-loss feedback during offline attack construction, without requiring knowledge of the defender's exact future tasks or deployment-time feedback.

For example, an attacker targeting a customer-support agent could obtain typical task patterns from public workflows and support-ticket templates, then use a surrogate agent to evaluate candidate memory edits through paired clean and poisoned executions.

Appendix H.4 also shows that feedback-free alternatives can produce harmful edits, while paired CMR feedback enables stronger selection.

To examine whether such attacks remain effective beyond the source-task distribution, we selected memory edits using source-family feedback only and evaluated them on unseen target families. We included matched source-only ASR-guided and feedback-free controls, without target-family feedback or tuning (Table W2).

Table W2. New leave-one-family-out evaluation: CMR on unseen task families

BenchmarkTarget tasksFeedback-free heuristicSource-only ASR-guidedSource-only MemHarmMemHarm − ASR (95% CI)Applicability (%)
AgentDojo520.2510.3260.392+0.066 [+0.012, +0.121]82.7
τ³-bench440.1650.2020.239+0.037 [-0.014, +0.087]84.1

MemHarm achieves the highest held-out CMR on both benchmarks under task-family shift, outperforming source-only ASR-guided and feedback-free selection without target-family feedback or tuning. These results support the transferability and practical value of severity-guided selection beyond the optimization task distribution. We will clarify the feedback assumptions and distribution-shift evaluation in the revised paper.

Summary and Planned Revisions

The new independent security-consequence audit provides direct evidence of higher unauthorized-action rates and consequence-sensitive regret under CMR guidance. Our source-only transfer experiments further show that MemHarm achieves the highest held-out CMR on both benchmarks under task-family shift, without target-family feedback.

We will clarify the operational interpretation of CMR, the practical assumptions underlying offline feedback, and incorporate both evaluations and their statistical evidence into the revised paper. We appreciate the reviewer's constructive suggestions, which have helped us strengthen the empirical evidence and clarify the practical relevance of our approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.