- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper tackles the issue of prompt injection from a potentially one-time event leading to a persistent risk because of memory poisoning. The authors propose a benchmark, MemoryDojo that evaluates how malicious instructions injected into an LLM agent’s tool observation can persist in memory, propagate through memory updates, and influence the agent’s behavior in later sessions.
Critical strengths
(1) The paper is well-motivated with a critical security issue and easy to follow. (2) The separation of insertion, retrieval, persistence, propagation and downstream distinct and identifies where a memory architecture or defense fails. (3) The benchmark supports five memory systems representing different approaches and broad coverage of task settings.
Critical weaknesses
(1) The five memory systems use different configurations, such as retrieval budget. Contributing the poisoning effect to memory architecture is not well analyzed or supported. (2) LLM judge needs further validation for reliability of measurements. (3) The proposed optimizer is evaluated with a comparison with strong baselines.
Questions for authors
The weaknesses all deserve attention and clarification.
We thank the reviewer for their attentive reading and positive feedback on our work.
W1 Retrieval budget ablation
At an approximately equal retrieval budget of around 800 tokens for each memory system, we present the results for the single-domain trajectories using optimized poison.
At this roughly equal retrieval budget, we still observe variation in BRT and the lifecycle metrics.
W2 Judge reliability
To reduce the reliance on LLM judge, we first perform deterministic exact matching to identify if the full poison text can be found in memory, which accounts for a large portion of the cases. The LLM judge then performs judgement for non-exact cases.
To validate this, we conduct a human validation and separate AI audit with a different model. Both human annotators and AI reviewer agreed with the original judge on whether poison content is either inserted or retrieved. We drew 100 samples containing 50 insertion and 50 retrieval occurrences. Two human annotators reviewed these decisions, reaching 97% and 95% agreement with the original judgment. We also used a stronger model (GPT-5.6-Sol) than the original GPT-5.6-Luna judge to perform separate review, and obtained 96% agreement with the luna judgements. We will add these details to the paper.
W3 Optimizer ablation
We run additional ablation study experiments to isolate the benefits of the memory-aware aspect of optimization. We remove the memory-aware description and lifecycle feedback, keeping behavioral reach metric feedback and run optimization following the same budget as for the memory-aware optimization. The table below shows a comparison between single-domain optimized runs on GPT-5.6-Luna using poison prompts obtained from behavioral only versus memory-aware optimization.
From the results we can see that including the memory description and lifecycle feedback can improve BRT by around 5% and trajectory level reach by 14%. This demonstrates the benefits of the memory-aware aspect of the optimizer. We will add these results to the paper.