- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MEMSHIELD, a defense framework for persistent-memory poisoning in LLM agents, with a particular focus on attacks where multiple individually benign-looking memories jointly trigger malicious behavior. The method uses provenance/dependency graphs and counterfactual replay to identify Minimal Triggering Sets (MTS) and Minimal Blocking Sets (MBS), then quarantines selected memories and re-runs retrieval and planning to verify that the attack does not reappear through replacement memories.
Critical strengths
- The paper targets an important and underexplored threat model in which multiple memories jointly form an attack trigger that cannot be detected reliably through single-memory attribution.
- The combination of set-level counterfactual diagnosis with re-retrieval and re-planning is well motivated because quarantining one memory can change retrieval and expose alternative poisoned memories.
- The MTS/MBS formulation provides a useful conceptual distinction between memories sufficient to trigger harmful behavior and memories whose removal is sufficient to suppress it.
Critical weaknesses
- The empirical comparison is difficult to interpret because several baselines use different or adapted protocols, some method–dataset pairs are missing, and the paper explicitly avoids matched cross-protocol causal comparisons or significance testing.
- The main end-to-end evaluation uses a single protected model and a single reported seed, while the claimed cross-architecture transferability in the abstract is not supported by completed matched cross-model results in the current experimental export.
- Several advertised components, including the multi-agent debate planner, long-term experience reuse, and persistent cleanup, are not directly validated by matched end-to-end experiments, making the contribution of the full framework unclear.
- The framework appears computationally expensive, requiring potentially hundreds of replay calls and over one million tokens per diagnosis, yet the paper does not provide a matched latency or total-overhead comparison against baselines.
Questions for authors
-
Can you provide matched experiments across multiple protected LLMs to support the claimed cross-architecture transferability?
-
Why are the baseline comparisons not run under a common protocol and identical evaluation budget?
-
How much do the multi-agent planner and long-term experience store actually improve safety or reduce diagnosis cost?
-
What is the practical latency and token overhead of MEMSHIELD per triggered case compared with existing defenses?