- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MemPA, an interactive benchmark for on-policy evaluation of whether LLM-based personal assistants learn how a user wants to be assisted (i.e., their interaction preferences) and apply this knowledge across contexts and as preferences evolve over time. Rather than relying on static, pre-generated conversation histories as most existing memory benchmarks do, MemPA has the assistant interact with simulated users, so that the evaluated system’s own responses shape the interactions it must learn from.
Critical strengths
-
The paper tackles a well-motivated problem by proposing an interactive, on-policy benchmark for interaction preferences, which is central to the experience of long-term personal assistants.
-
The paper provides a fine-grained evaluation and an informative analysis of existing memory systems. Rather than relying on a single end-to-end score, MemPA evaluates assistants at two levels: preference selection (PSA, CS, PET) and realization, which is measured both behaviorally (BRS) and from the user’s perspective (PRS). Based on this evaluation, various findings about the existing memory systems are derived offering useful insights for future memory design.
Critical weaknesses
- The diversity of users covered by the benchmark is limited. It trades breadth for depth: each of the four persona-grounded users goes through 99 on-policy sessions with controlled preference changes, which is a substantially longer horizon than prior benchmarks. However, with only four distinct users, it remains unclear whether the findings generalize to users with different personas and preference profiles.
Questions for authors
-
Given additional compute and time, how easily could the number of simulated users be increased? For instance, is the construction of new users (personas, preference settings, and scheduled preference changes) and their corresponding probes automated, or does it require substantial manual effort?
-
Context appears to be limited to a binary work/personal distinction. Can the benchmark be extended to finer-grained contexts?
-
Since File memory is nanobot’s default memory, is it disabled when the other memory systems are evaluated? How would the results change if Mem0, MemOS, or Simple RAG were used on top of File memory, as would likely happen in a realistic nanobot deployment?
4 comments
Revised Manuscript
Dear Reviewer xPjS,
We have uploaded a revised manuscript, with all changes marked in blue. Your comments motivated several additions that we believe strengthen our work:
- Scalability. Appendix C.7 now describes how new users are added: which steps of the construction pipeline are automated, where the manual quality gate lies, and how finer contexts are set up without changing the taxonomy.
- Coverage of the four users (general response). Section 3.2 and Appendix B.1 now elaborate on the coverage and validity for the four users.
- Human validation of BRS and PRS. We have included the human validation in Section 4 (Simulator and Judges) , and Appendix H reports in details the agreement of three annotators with both LLM-based scores (Table R3).
To make reading easier, new tables are temporarily numbered as R1–R4, so all other table numbers stay as cited in our responses. Appendix F.4, cited in the general response, is now Appendix F.3.
As the discussion phase is approaching its end, we are happy to provide any further details. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score.
Thank you for your review!
Dear Reviewer XPjS,
We greatly appreciate your thoughtful comments on user diversity and benchmark scalability. They prompted us to better explain the extensibility of our evaluation pipeline.
The discussion period will end in a few hours. In our responses and revised manuscript, we detailed the psychological coverage of our four users (Table R1) and our ~95% automated pipeline for adding new users and finer-grained contexts (Appendix C.7). We also clarified our isolation setup for evaluating memory architectures.
We would like to know if these updates have adequately addressed your questions. We would be happy to answer any remaining questions before the deadline.
Sincerely, The Authors
Thank you for the detailed response and revisions. I appreciate the additional analysis of preference coverage, clarification of the construction pipeline, and human validation results.
While the four users provide meaningful coverage of preference settings, my main concern about generalizability remains. All four are derived from two sitcoms, leaving it unclear whether the findings extend to different user profiles and interaction patterns. Evaluation with additional users grounded in more diverse sources, including real-world interaction data, would further strengthen the conclusions.
I also understand the rationale for evaluating memory systems in isolation. However, since File memory is nanobot's default, it remains unclear whether the observed retrieval limitations would persist in more realistic combined configurations.
Overall, I recognize the benchmark's value and appreciate the authors' efforts. Given these remaining concerns, I will maintain my rating.
Response to Reviewer xPjS
We thank the Reviewer for recognizing the "well-motivated problem" and the "fine-grained evaluation" of existing memory systems. We appreciate your questions on scalability, which is the issue we care about most as well. We respond to each point below.
Point 1. With only four users, do the findings generalize to users with different personas and preference profiles?
Please refer to the general response.
Point 2. How easily can more users be added? Is constructing personas, preference settings, preference changes, and probes automated?
Good question. Adding a user is mostly automated, and the manual work is a small quality gate.
For a new character with existing transcripts, our pipeline extracts the 28 preference cells with their evidence, writes the persona card, anonymizes the names, and drafts the world. Then, based on the world, it plans the 94 accumulation sessions, the 5 change events, and the probes. Rule-based validators and LLM semantic checks send back drafts that break the rules, and the pipeline iterates.
The manual part is deciding the few cells whose transcript evidence is thin or conflicting, and spot-checking the world and the rejected edge cases. We keep this gate on purpose. It is what lets us trust that preferences are conveyed accurately and not leaked.
Point 3. Context is a binary work/personal split. Can the benchmark be extended to finer-grained contexts?
Yes. With the automated pipeline above, contexts can be split more finely. The taxonomy of interaction preferences does not change. We only need to multiply the contexts when we set up a persona, so the environment is easy to configure.
Point 4. Is File memory disabled when the other memory systems are evaluated, and how would results change if they ran on top of File memory?
Yes, File memory is turned off when we evaluate the other memory systems. As our results show, Mem0, MemOS, and Simple RAG remain at the No Memory level (PSA of 50.0%, 50.0%, and 49.9% vs. 49.6%). We evaluated each system in isolation to avoid complex engineering entanglement and to cleanly compare the core extraction and retrieval logic.
But this point is valid and interesting. We are truly excited by future work, inspired by our work, that can work out combined settings or even more memory architectures specifically tailored to human-agent interaction performance.
We hope these responses and the new results address your concerns! If they do, we would appreciate it if you would consider updating your score, and we are happy to discuss any remaining questions.