- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MemSecBench is a task-grounded agent memory security benchmark that traces malicious semantics through a Write–Execute–Forget lifecycle with evidence-based checkpoints and matched comparisons across agent harnesses, LLM backends, and memory backends.
Critical strengths
Existing evaluations often test only whether malicious content is written or recalled, while MemSecBench links memory persistence to a verified external consequence and selective repair. Its Write, Execute, Forget framing, separate Execute and Forget evaluation from the same post-Write state, benign-memory preservation emphasis, cross-backend and cross-harness comparison, and human validation of checkpoint judgments together make a meaningful step beyond single-system memory-poisoning studies.
Critical weaknesses
The reliability of the repair checkpoint is concerning: in the human-validation sample, 24 of 64 repair failures were incorrectly classified as successes. Since selective repair is a central evaluation dimension, the authors should clarify how these adjudication errors affect the reported repair success rates and cross-backend comparisons. The manuscript also lacks sufficient editorial polish, with overly procedural, step-by-step descriptions that weaken the clarity of its methodological presentation.
Questions for authors
How sensitive are the reported selective-repair results to the 24 false-positive repair judgments observed in the human-validation sample?
3 comments
Dear Reviewer DW57,
Thank you for your careful review and constructive feedback on our paper. We have posted a point-by-point response to the issues you raised.
We would appreciate it if you could take a look at our response and let us know whether any questions or concerns remain. We would be happy to clarify any points or discuss them further. Thank you again for the time and effort you have put into reviewing our submission.
Best regards, The authors
Thank you for the clarification and audit analysis. I will keep my rating unchanged.
Thank you for your careful review.
Q1. Sensitivity of selective repair rates to F1 false positives
Our automated judge makes errors at several checkpoints, including repair. Using the 500-run audit in Table 12, we compared pass rates under the automated judgments and the adjudicated human labels.
The automated judge agreed with the human reference on 90.70% of F1 judgments and 88.84% of F2 judgments. In this audit, the net change in the aggregate pass rate was below 3 percentage points at every checkpoint, with a maximum absolute change of 2.40 percentage points.