- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper measures memory misuse in LLM agents: when a memory influences the agent response beyond what the user request allows. MemMisBench, a 1800 item benchmark, is constructed to measure models and memory systems across three forms of misuse: evidence, authority, and permission. Results show that the presence of memories raises misuse rates from a 14% baseline to 62.1% averaged across 7 models and five memory systems. The authors attribute memory misuse to a lack of licensing judgement rather than retrieval relevance of the memories. The core contribution is the benchmark, which allows for matched controls to isolate the effects of unlicensed memories. LLM-judge results were validated against human labelers with an agreement of 0.90
Critical strengths
- The evaluation scope is broad and diverse across a range of seven models and five memory systems, showing that memory misuse risks are not isolated to a specific model or memory setup in the given setting. Judging is also human-validated with a high agreement of 0.90
- The paper presents a useful distinction between memory relevance and whether memory usage is licensed.
Critical weaknesses
- The paper's experiments do not properly isolate memory licensing judgement. Comparisons are primarily drawn between misuse rates when unlicensed memories exist versus don't exist. This fails to show that the misuse is due to a failure of licensing judgment. A concrete experiment would be to fix the memory set but vary the user query contents to request vs not request the unlicensed memory item.
- The realism of the benchmark is unclear and relevance to production deployment of memory-enhanced agents is not discussed. The 1800 benchmark items are synthetically generated and limited in size (9-10 records), raising concerns on whether the unlicensed memory would be retrieved in a realistic deployment environment. Focal memories were deliberately constructed to induce specific misuse patterns, making it unclear whether these scenarios match realistic memory distributions.
- The unified taxonomy is presented as a central contribution but is not motivated nor used. The introduction says the three roles (evidence misuse, authority misuse, permission misuse) were motivated by analysis but no such analysis is described. Results are broken down by each of the six categories but the roles are never used to guide further analysis.
Questions for authors
No specific questions beyond what is stated in the weaknesses