- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies poisoning of persistent natural-language memory in LLM agents. Its central observation is that a malicious influence may be distributed across several memories that look benign in isolation and becomes dangerous only after retrieval composes them into a security-sensitive dependency.
Critical strengths
The choice of analysis granularity is well motivated. Treating individual typed relations, rather than whole memory records, as the units of provenance and validation is well matched to attacks whose components are innocuous separately. Re-running the analysis on the joint graph at retrieval time is consequential because it targets dependencies that do not exist when memories are ingested independently.
Critical weaknesses
The evaluation is not specified well enough to support the comparative claims. The paper omits attack-generation details, AgentDojo denominators, benign-set size, baseline calibration, and an end-to-end measure of whether sensitive disclosure or unauthorized action was actually prevented. The reported values are also difficult to reconcile: unauthorized-access and fact-tampering results differ between Tables 1 and 2, and MINJA appears with 45.1%, 85.5%, and 87.5% across insufficiently distinguished settings.
Questions for authors
- What are the exact denominators, benign-set sizes, and undefended attack-success rates for every AgentDojo result and for the stated 298 evaluation units?
- Why do the full-MELT values in Tables 1 and 2 differ for apparently corresponding attacks, and what precisely distinguishes the 45.1%, 85.5%, and 87.5% MINJA settings?
- Can the authors report end-to-end attack success and benign task success before and after enforcement, rather than detection rate alone?
3 comments
We would greatly appreciate any further feedback or remaining concerns you may have, and we would be more than happy to make additional revisions to address them.
Thanks for the detailed rebuttal. I maintain my score to support this paper.
We thank the reviewer for the detailed evaluation questions. Q1: Evaluation denominators and attack construction. We added details of the DeepSeek-V4-based attack-generation procedure, covering target-question construction, poisoned-memory generation, adversarial-prompt generation, and memory-formation validation (Appendix A.5). Each of the five non-MINJA AgentDojo attack families comprises 164 original cases. We also clarified the distinction between the 250-episode calibration set and the separate 86-task benign evaluation. The previously reported 298 refers to entities rather than attack evaluation units. Figure 2 now presents ASR with and without defense across all six attack families, while Table R1 reports detailed attack-family results.
Table R1. Attack success rates (ASR), detection rates (DR), and extraction failure rates (EFR) across controlled and AgentDojo environments.
Q2: Differences between evaluation settings. We clarified that Table 1 uses separate controlled ablation subsets while maintaining the same models, validation configuration, and evaluation metrics as the main experiments (Appendix A.5). The MINJA detection rates of 45.1% and 85.5% correspond to the controlled benchmark and AgentDojo, respectively (Table R1). Differences in entity-reference clarity may affect dependency extraction. The previously reported AgentDojo detection rates of 85.5% and 87.5% resulted from experimental variation across independent runs under identical settings. To avoid ambiguity, we standardized the MINJA detection rate to 85.5% in Sec. 5.3 and Figure 3, ensuring consistency with the main evaluation. The ablation results use separately selected controlled subsets (Table 1 in the revised manuscript), which explains the differences between their detection rates and those reported in Table R1.
Table R2. Benign task success before and after enabling MELT on AgentDojo.
Q3: End-to-end outcomes and benign utility. We now report ASR with and without MELT across all six attack families (Table R1), evaluating whether poisoned memories induce attacker-targeted harmful instructions or commands in subsequent agent interactions. On AgentDojo, MELT reduces the macro-average ASR from 82.7% to 12.2%, while benign task success declines by only 3.49 percentage points across 86 legitimate tasks (Table R2). These results demonstrate that MELT substantially mitigates harmful outputs while largely preserving benign task performance.