acceptodds
Under review as a conference paper at ICLR 2027

Why Bother Me Again? Benchmarking Personal Assistant Memory for Interaction Preferences

Abstract

Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires assistants to remember how users want to be assisted, but existing memory benchmarks focus on factual recall and personalized answers and rarely test how assistants learn and act on interaction preferences. We introduce \mempa, an interactive environment for on-policy evaluation of memory in personal assistants. \mempa poses long-horizon tasks requiring assistants to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG), do not outperform a memoryless assistant, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the assistant's default interaction habits remains challenging. The environment and dataset are available.

Primary area:
datasets and benchmarks
Keywords:
Agent, Personal Assistant, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xPjSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemPA, an interactive benchmark for on-policy evaluation of whether LLM-based personal assistants learn how a user wants to be assisted (i.e., their interaction preferences) and apply this knowledge across contexts and as preferences evolve over time. Rather than relying on static, pre-generated conversation histories as most existing memory benchmarks do, MemPA has the assistant interact with simulated users, so that the evaluated system’s own responses shape the interactions it must learn from.

Critical strengths
  1. The paper tackles a well-motivated problem by proposing an interactive, on-policy benchmark for interaction preferences, which is central to the experience of long-term personal assistants.

  2. The paper provides a fine-grained evaluation and an informative analysis of existing memory systems. Rather than relying on a single end-to-end score, MemPA evaluates assistants at two levels: preference selection (PSA, CS, PET) and realization, which is measured both behaviorally (BRS) and from the user’s perspective (PRS). Based on this evaluation, various findings about the existing memory systems are derived offering useful insights for future memory design.

Critical weaknesses
  1. The diversity of users covered by the benchmark is limited. It trades breadth for depth: each of the four persona-grounded users goes through 99 on-policy sessions with controlled preference changes, which is a substantially longer horizon than prior benchmarks. However, with only four distinct users, it remains unclear whether the findings generalize to users with different personas and preference profiles.
Questions for authors
  1. Given additional compute and time, how easily could the number of simulated users be increased? For instance, is the construction of new users (personas, preference settings, and scheduled preference changes) and their corresponding probes automated, or does it require substantial manual effort?

  2. Context appears to be limited to a binary work/personal distinction. Can the benchmark be extended to finer-grained contexts?

  3. Since File memory is nanobot’s default memory, is it disabled when the other memory systems are evaluated? How would the results change if Mem0, MemOS, or Simple RAG were used on top of File memory, as would likely happen in a realistic nanobot deployment?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xPjS

We thank the Reviewer for recognizing the "well-motivated problem" and the "fine-grained evaluation" of existing memory systems. We appreciate your questions on scalability, which is the issue we care about most as well. We respond to each point below.


Point 1. With only four users, do the findings generalize to users with different personas and preference profiles?

Please refer to the general response.


Point 2. How easily can more users be added? Is constructing personas, preference settings, preference changes, and probes automated?

Good question. Adding a user is mostly automated, and the manual work is a small quality gate.

For a new character with existing transcripts, our pipeline extracts the 28 preference cells with their evidence, writes the persona card, anonymizes the names, and drafts the world. Then, based on the world, it plans the 94 accumulation sessions, the 5 change events, and the probes. Rule-based validators and LLM semantic checks send back drafts that break the rules, and the pipeline iterates.

The manual part is deciding the few cells whose transcript evidence is thin or conflicting, and spot-checking the world and the rejected edge cases. We keep this gate on purpose. It is what lets us trust that preferences are conveyed accurately and not leaked.


Point 3. Context is a binary work/personal split. Can the benchmark be extended to finer-grained contexts?

Yes. With the automated pipeline above, contexts can be split more finely. The taxonomy of interaction preferences does not change. We only need to multiply the contexts when we set up a persona, so the environment is easy to configure.


Point 4. Is File memory disabled when the other memory systems are evaluated, and how would results change if they ran on top of File memory?

Yes, File memory is turned off when we evaluate the other memory systems. As our results show, Mem0, MemOS, and Simple RAG remain at the No Memory level (PSA of 50.0%, 50.0%, and 49.9% vs. 49.6%). We evaluated each system in isolation to avoid complex engineering entanglement and to cleanly compare the core extraction and retrieval logic.

But this point is valid and interesting. We are truly excited by future work, inspired by our work, that can work out combined settings or even more memory architectures specifically tailored to human-agent interaction performance.


We hope these responses and the new results address your concerns! If they do, we would appreciate it if you would consider updating your score, and we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Revised Manuscript

Dear Reviewer xPjS,

We have uploaded a revised manuscript, with all changes marked in blue. Your comments motivated several additions that we believe strengthen our work:

  • Scalability. Appendix C.7 now describes how new users are added: which steps of the construction pipeline are automated, where the manual quality gate lies, and how finer contexts are set up without changing the taxonomy.
  • Coverage of the four users (general response). Section 3.2 and Appendix B.1 now elaborate on the coverage and validity for the four users.
  • Human validation of BRS and PRS. We have included the human validation in Section 4 (Simulator and Judges) , and Appendix H reports in details the agreement of three annotators with both LLM-based scores (Table R3).

To make reading easier, new tables are temporarily numbered as R1–R4, so all other table numbers stay as cited in our responses. Appendix F.4, cited in the general response, is now Appendix F.3.

As the discussion phase is approaching its end, we are happy to provide any further details. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Thank you for your review!

Dear Reviewer XPjS,

We greatly appreciate your thoughtful comments on user diversity and benchmark scalability. They prompted us to better explain the extensibility of our evaluation pipeline.

The discussion period will end in a few hours. In our responses and revised manuscript, we detailed the psychological coverage of our four users (Table R1) and our ~95% automated pipeline for adding new users and finer-grained contexts (Appendix C.7). We also clarified our isolation setup for evaluating memory architectures.

We would like to know if these updates have adequately addressed your questions. We would be happy to answer any remaining questions before the deadline.

Sincerely, The Authors

Thank you for the detailed response and revisions. I appreciate the additional analysis of preference coverage, clarification of the construction pipeline, and human validation results.

While the four users provide meaningful coverage of preference settings, my main concern about generalizability remains. All four are derived from two sitcoms, leaving it unclear whether the findings extend to different user profiles and interaction patterns. Evaluation with additional users grounded in more diverse sources, including real-world interaction data, would further strengthen the conclusions.

I also understand the rationale for evaluating memory systems in isolation. However, since File memory is nanobot's default, it remains unclear whether the observed retrieval limitations would persist in more realistic combined configurations.

Overall, I recognize the benchmark's value and appreciate the authors' efforts. Given these remaining concerns, I will maintain my rating.

Reviewer_V5WiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work investigates whether personal assistant memory learns interaction preferences — how a user wants to be assisted, such as when to clarify, when to confirm before acting, and how proactively to reach out — rather than only facts and content. To address it, the authors proposed MemPA, an executable, on-policy benchmark in which four simulated users, each holding 28 preference settings, interact with an assistant over 99 sessions across four months, including five controlled preference changes, while evaluation separates failures into acquisition, selection, and realization against No Memory and Oracle reference conditions. Different from existing benchmarks that focus on factual recall and content personalization, MemPA tests whether preferences are learned, applied in the right context, and tracked as they change. Experimental results verify how limited current memory is: No Memory reaches 49.6% selection accuracy against 45.2% for chance, Mem0, MemOS, and Simple RAG stay at about 50%, and only File memory improves selection, to 59.2%.

Critical strengths
  • Failures are localized, not just scored. Preferences are formalized as a state indexed by user, context, attribute, and time, and the No Memory and Oracle reference conditions let the authors attribute a failure to acquisition, selection, or realization instead of reporting a single end-to-end number. This design is what makes the paper's diagnostic findings interpretable.
  • A genuinely on-policy, persistent-user protocol. Each memory condition builds its own history through 99 live sessions per user, including five controlled preference changes; no compared benchmark simultaneously covers executable tasks, preference-conditioned reactions, persistent on-policy histories, contextual isolation, and controlled change.
  • The "stored yet not retrieved" diagnosis is backed by direct input inspection. In sampled final probes, an applicable preference appears in the assistant's input in only 7 of 334 judged probes under the retrieval conditions, versus 60 of 111 (54%) under File memory, and a store-level case study confirms the preference was written to all three retrieval stores but retrieved by none.
Critical weaknesses
  • Half of the ground truth is author-adjudicated rather than transcript-derived, unevenly across users. Of the 112 preference cells, only 53 were confirmed at high confidence from transcript evidence and 23 were set by the reviewer; for Users C and D only 5 of 28 cells each are high-confidence (Table 5). The persona-grounded claim is therefore much weaker for those users, and no inter-rater reliability is reported for the adjudication step itself.
  • The simulator always emits an explicit, learnable signal, unlike real users. The reception side "leaves no room for silent acceptance": every session guarantees an explicit acceptance after correct behavior and a correction after wrong behavior, although the paper's own introduction notes real preferences are also revealed by a tolerated shortcut, i.e., silence. This makes the acquisition problem substantially cleaner than deployment, and the conveyance validation was run only on File-memory runs.
  • One model simulates the user and produces both realization scores. Gemini-3.8-Flash plays the user simulator and also gives both the judge-scored and the user-rated realization scores, so their reported agreement (ρ = 0.83) may be inflated by shared-model bias; the independent second judge agrees noticeably less with the user-rated score (ρ = 0.71 vs. 0.85). The conclusion that the agreement "mainly reflects the interaction itself" does not rule out the correlated-bias explanation.
  • The headline comparison is confounded twice over. File memory's consolidated file reaches the assistant in full under a ~20k-token prompt budget, whereas the retrieval conditions are capped at the top 10 results and at most 4,000 tokens, so the contrast conflates the memory mechanism with how much evidence reaches the assistant; and MemOS runs GPT-5.5 as its memory-side LLM while Mem0 and File memory use DeepSeek-V4-Flash, so cross-system differences cannot be attributed to architecture alone.
  • Several metrics rest on very few units, and the scope is narrow. Context sensitivity pools only 18 differing user–attribute pairs and change tracking only 20 changed cells, with correspondingly wide intervals — File memory's context-sensitivity difference (−7.4 [−17.1, +1.4]) includes zero (Table 13). All results rest on four fictional users from two U.S. sitcoms, a scoping the abstract and conclusion do not carry.
Questions for authors
  • Would the retrieval-based systems still fail if their evidence budget matched File memory's — a full consolidated file under a ~20k-token prompt versus top-10 results capped at 4,000 tokens? A budget-matched Simple RAG ablation seems necessary before attributing the gap to retrieval itself.
  • Do the headline results hold when selection accuracy, context sensitivity, and change tracking are computed only over the 53 high-confidence ground-truth cells, excluding the reviewer-set cells that dominate Users C and D?
  • Given that one model is simultaneously the user simulator and the source of both realization scores, and the independent judge agrees less, can you provide a human-rated subsample to anchor at least one of the two realization scores?
  • If the simulator sometimes stayed silent after a wrong behavior — the "tolerated shortcut" case — how much of File memory's selection advantage survives? Is there a variant of the benchmark with sparse feedback?
  • In the input-inspection analysis, 112 final probes were sampled per condition, yet the judged counts are 334 for the three retrieval conditions pooled (336 expected) and 111 for File memory. What happened to the other three probes, and were they excluded by a rule that could correlate with condition?

I'm willing to raise scores if my concerns can be addressed, the benchmark is of interests to the community.

5 comments
AuthorsOpenReview ↗

Response to Reviewer V5Wi (1/3)

Thank you for the careful reading, and for describing a “genuinely on-policy, persistent-user protocol” where “failures are localized”. Your questions point to several places where we can explain more clearly. We answer them below.


Point 1. Only 53 of 112 cells are high-confidence and 23 are set by the authors, mostly for Users C and D. Do the headline results hold on the high-confidence cells alone?

Thank you for looking closely at how the ground truth is grounded. We wanted to start by clarifying a key distinction. The confidence levels in Table 5 measure the auto-extractor's initial certainty, not the reliability of the final ground truth. Cells below high auto-confidence are the ones the authors verified directly against the transcripts, so their final settings are unambiguous. We do include a full paragraph in Appendix B.3 explaining this. We now better signpost that section in the main text.

  1. Auto-confidence only decides which cells trigger human review. During extraction, a cell gets high confidence automatically only when a character's explicit words and their reactions to others agree across scenes. User A states outright how he wants to be helped, so 27/28 cells pass automatically. Users C and D come from The Office, where workplace exchanges are indirect and plot-driven. When signals were subtle or competing, our pipeline conservatively lowered the auto-confidence so a human would step in.

  2. Human review resolves ambiguity from transcript evidence. Two authors, who both know the preference definitions well and calibrated on them beforehand, adjudicated the cells below high auto-confidence against their cited scenes (walk-through in Appendix B.5). They either confirmed the extractor's proposal (34 Reviewed & Kept cells) or resolved competing readings of the scenes (18 of the Set by author cells). Only 7 cells had no transcript evidence and were set manually. They are in attributes such as Capability Boundary, which rarely come up in conversation between people. All the other 105 of the 112 cells are grounded in cited transcript scenes. The two authors decided each cell jointly and discussed every disagreement until they agreed.

The 53 high-confidence cells are the strictest subset during the extraction process, not the only reliable one. Based on your suggestion, we recomputed PSA, CS and PET on the 87 cells where the extractor's setting was kept, leaving out the 25 set by author cells. Values are averages over the four backbones (%).

PSACSPET
MemoryAll 11287 cellsAll 11287 cellsAll 11287 cells
File memory59.256.214.413.225.424.5
Simple RAG49.947.020.816.715.015.2
Mem050.046.822.222.914.214.2
MemOS50.047.219.922.210.810.3
No Memory49.647.021.822.211.211.3
Oracle99.099.099.5100.098.398.5

The findings hold on the subset. File memory still leads in PSA and PET, and its CS stays below No Memory. The subset keeps 12 of 18 CS pairs and 17 of 20 PET items, so the CS values move more than the others.

AuthorsOpenReview ↗

Response to Reviewer V5Wi (2/3)

Point 2. The simulator always gives explicit feedback, while real users may stay silent. How much of File memory's advantage survives sparse feedback, and does conveyance hold beyond File-memory runs?

We appreciate this motivating point! Of course, people may respond with sparse feedback - and that's a worthwhile question to assess. However, the explicit feedback is partly by design! Each piece of feedback is meant to reach the memory, as a controlled and reproducible diagnostic. If feedback were sparse and ambiguous, we could not tell whether a memory system failed at retrieval or never received the signal. Therefore, the reception side has to be explicit. Nonetheless, this is indeed an interesting extension. For example, we could model how differently users give feedback and how that changes what a PA learns over long interaction, which should vary widely across people!

Regarding the conveyance check, we checked only File memory because the simulator keeps the one and only reception rule regardless of the PA's memory, so that result should be representative. However, we believe adding the other memory systems, as you suggested, would make the results more complete and sound. So here we provide the conveyance check for all four memory systems.

As in Appendix F.9, two human annotators read 112 accumulation sessions per memory system (448 in total, 28 per user) and judged whether a reader seeing only the visible conversation can identify the target setting from the user's reactions; results averaged over both annotators are below.

MemorySessionsConveyed
File memory11299.1%
Simple RAG11299.1%
Mem011298.2%
MemOS11298.7%

The preference is conveyed at essentially the same rate (98.2% to 99.1%) across memory systems.


Point 3. One model plays the user and produces both realization scores, which may inflate their agreement. Can a human-rated subsample anchor at least one of them?

Thank you for calling attention to the evaluation validity! We agree that a human check of BRS and PRS on a subset of probes would better anchor the metrics. As you suggested, we organized and ran a human validation as below.

Three annotators (AI/HCI researchers, with undergraduate degree or above) each scored 112 final probes, 28 per user and 28 per memory system, with the exact same BRS rubric and the PRS question that was given to LLM judge. A lead researcher first ran a calibration session to align their standards. Each annotator then scored independently, without seeing any model scores or the others' ratings. Each annotator agrees closely with both LLM-based scores, and less with the second judge:

Annotator 1Annotator 2Annotator 3
ComparisonρExactρExactρExact
Human BRS vs. judge BRS (Gemini)0.9583.9%0.9482.1%0.9279.5%
Human PRS vs. simulator PRS (Gemini)0.9483.9%0.9482.1%0.9378.6%
Human BRS vs. second-judge BRS (Claude Sonnet 5)0.8369.6%0.8368.8%0.8673.2%

Each Gemini score therefore tracks the human ratings on its own. We have added the result to the paper.

AuthorsOpenReview ↗

Response to Reviewer V5Wi (3/3)

Point 4. The headline comparison may be confounded by unequal evidence budgets (the full consolidated file vs. top-10 results capped at 4,000 tokens) and by MemOS using GPT-5.5 as its memory LLM.

Excellent point!

Evidence budget. The ~20k-token limit is the shared compaction threshold for raw conversation across all conditions, not the size of MEMORY.md. To verify the actual evidence budget, we measured the exact token counts reaching the PA's prompt across all final probes.

MemoryMemory text (Med / Mean)Memory + Conv. (Med / Mean)
File memory4.6k / 5.5k4.8k / 5.5k
Simple RAG3.2k / 3.0k5.0k / 4.9k
Mem00.6k / 0.6k1.6k / 2.2k
MemOS0.9k / 0.8k2.2k / 2.5k

This reveals three facts:

  1. Actual token volumes are comparable, not 20k vs. 4k. File memory averages only 4.6k–5.5k tokens. Because retrieval conditions also retain 1.0k–2.2k tokens of raw conversation, the total context reaching the PA is virtually identical between File memory (4.8k median) and Simple RAG (5.0k median).
  2. Mem0 and MemOS never hit the 4k cap. Their top-10 retrieved memories average under 1k tokens. Raising the 4,000-token cap would not add a single token to their prompts.
  3. The bottleneck is relevance, not volume. Despite receiving similar total context (4.8k vs. 5.0k), File memory outperforms Simple RAG by 9.3 points of PSA. As Finding 2 shows, retrieval surfaces mostly task content, returning an applicable interaction preference in only 7 of 334 probes versus 60 of 111 for File memory.

Memory LLM. Because MemOS's default memory LLM (gpt-4o-mini) was unavailable when we set up, we ran it with GPT-5.5. However, we agree that a shared memory LLM would make the comparison cleaner. Following your suggestion, we reran MemOS with DeepSeek-V4-Flash as its memory LLM across all four users (paired with the DeepSeek-V4-Flash PA). We plan to include all reruns and update our paper accordingly; but wanted to share the bulk of our response to you now. The results confirm that our finding still holds.

MemOS memory LLMUser AUser BUser CUser DAvg.
PSA, GPT-5.5 (paper)40.554.842.963.150.3
PSA, DeepSeek-V4-Flash39.357.146.460.750.9
BRS, GPT-5.5 (paper)62.970.554.570.064.5
BRS, DeepSeek-V4-Flash62.968.662.962.964.3

Point 5. CS and PET rest on few units (18 pairs, 20 changed cells), File memory's CS interval includes zero, and the abstract and conclusion do not state the four-user sitcom scope.

We would like to clarify a possible misreading. The CS interval including zero supports Finding 5. Finding 5 does not claim that File memory changes CS. It claims that no memory system improves CS (“CS and PET stay low under every memory condition”). File memory’s CS difference is −7.4 [−17.1, +1.4]. The interval spans zero, so the difference is not significant and File memory brings no gain in CS. That is what Finding 5 says.

Regarding the few units for CS and PET, we wish to point out this is because both phenomena are sparse. A real user rarely holds separate standards for every interaction attribute across work and personal life. So 18 of the 56 user–attribute pairs (32%) differ across contexts. Likewise, users do not flip all their habits over four months, so we model five event-driven shifts per user (20 changing cells in total).

However, we agree that this limitation can be better noted in the writing. We will state the scope in a Limitations section. We address the number of users question jointly for all reviewers in the general response.


Point 6. The input-inspection counts are three probes short (334 of 336 pooled, 111 of 112 for File memory). What happened to them, and could the exclusion correlate with condition?

Thank you for checking the counts. We re-checked and found that the three probes were dropped because the readers, who were blind to the memory condition, could not tell whether the retrieved memory applied to the target setting. For example, one RAG input said “...they got annoyed when I added commentary. Keep it clean...”, which could mean emotional acknowledgment or only length. So these three were discarded. Since the readers did not know the condition, this rule cannot favor any condition. Adding the three back as hits or as misses changes each rate by less than one percentage point.


We hope these responses and the new results address your concerns! If they do, we would appreciate it if you would consider updating your score, and we are happy to discuss any remaining questions.

Reviewer_RckCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemPA, a benchmark for evaluating whether personal assistants can learn and apply users' interaction preferences over long-term interactions. The benchmark features four simulated users, each participating in 99 sessions over four months, with 14 interaction preference attributes across work and personal contexts and controlled preference changes. The authors evaluate four memory systems across four LLM backbones using metrics that assess preference selection, context sensitivity, preference evolution, and behavioral realization. The results show that retrieval-based memory systems offer little improvement over a memoryless baseline, while file-based memory improves preference selection but struggles with contextual distinctions, and even explicitly providing the correct preferences does not always ensure appropriate behavior.

Critical strengths
  • Clear Problem Formulation and Fine-Grained Evaluation: The paper provides a clear formulation of interaction preferences and decomposes memory-based personalization into preference acquisition, selection, and behavioral realization. I particularly appreciate that the evaluation goes beyond preference recall by introducing metrics for context sensitivity (CS) and preference evolution tracking (PET), allowing the analysis of different failure modes. The distinction between selecting the correct preference and actually following it in the delivered interaction is also valuable, as it helps identify where personalization fails.
  • Comprehensive and Well-Structured Interactive Evaluation Environment: The paper presents a well-structured executable environment for evaluating memory over long-horizon interactions. The benchmark incorporates 99 sessions per user, persistent world states, 83 executable tools across 26 domains, and controlled preference-change events. Importantly, the evaluated agents accumulate their own interaction histories rather than relying solely on preconstructed conversations. The use of isolated evaluation probes, checkpoint-based testing, and repeated runs also provides a systematic protocol for evaluating memory behavior over time.
Critical weaknesses
  • Limited Novelty and Insufficient Comparison with Related Work: While the paper addresses an important problem, I am not fully convinced of its novelty compared to existing benchmarks for personalized agents. Several closely related works, such as ProPerSim (Kim et al., 2026), UserVille (Sun et al., 2025), and KnowU-Bench (Chen et al., 2026), have already explored personalized interaction behaviors, user simulation, preference learning from feedback, and proactive assistance, yet they are not adequately discussed or compared in the paper. Although MemPA combines long-horizon memory, contextual preferences, and preference changes, it remains unclear what fundamentally new evaluation challenges or insights this combination introduces beyond prior work. A more comprehensive comparison with these studies would help clarify the unique contributions of MemPA.
  • Lack of Human Validation for LLM-Based Evaluation: The benchmark relies on LLM-based judges for its behavioral realization (BRS) and perceived realization (PRS) metrics, but their agreement with human judgments is not evaluated. Although the authors report strong agreement between BRS and PRS and conduct additional cross-validation with another LLM judge, these results do not necessarily establish that the metrics reflect human perceptions of interaction quality. This is particularly important because the evaluated preferences involve subjective aspects of communication, such as verbosity, emotional engagement, and autonomy. While human annotators validate the consistency of simulated user feedback, a human evaluation of a representative subset of agent interactions would provide stronger evidence for the reliability of the proposed evaluation metrics.
  • Limited Persona Diversity and Potential Data Contamination: The benchmark relies on only four personas derived from two popular American television series, raising concerns about the diversity and generalizability of the evaluation. It is unclear why these particular personas were selected or whether they sufficiently represent diverse interaction preferences. Furthermore, Table 5 shows substantial variation in ground-truth confidence, with 27 of 28 preference cells confirmed at high confidence for User A, compared to only 5 for each of Users C and D. The use of well-known fictional characters also raises potential pretraining contamination concerns. Although the authors conduct an anonymization-based shortcut check, it remains unclear whether models could recognize the original characters from their behavioral patterns over repeated interactions. Evaluations with more diverse, independently constructed personas would strengthen the benchmark's validity.

References

  • Kim, J., et al. (2026). ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation. International Conference on Learning Representations (ICLR).
  • Sun, W., et al. (2025). Training Proactive and Personalized LLM Agents. arXiv preprint arXiv:2511.02208.
  • Chen, T., et al. (2026). KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation.arXiv preprint arXiv:2604.08455.
Questions for authors
  • Novelty and Related Work: Could the authors clarify how MemPA differs from closely related benchmarks such as ProPerSim (Kim et al., 2026), UserVille (Sun et al., 2025), and KnowU-Bench (Chen et al., 2026)? In particular, what new evaluation challenges or insights does MemPA introduce that are not adequately captured by these existing benchmarks?
  • Human Validation of Evaluation Metrics: Have the authors considered validating BRS and PRS against human judgments? Given the subjective nature of interaction preferences, could the authors provide human annotations on a representative subset of probes and report their agreement with the LLM-based scores?
  • Persona Selection and Data Contamination: What motivated the selection of only four personas from two TV series, and how representative are they of diverse interaction preferences? Additionally, given the substantial variation in ground-truth confidence across personas (Table 5), have the authors examined whether the main findings are sensitive to annotation confidence? Could the authors also provide evidence that the anonymized personas cannot be re-identified from their interaction histories, potentially allowing models to exploit pretrained knowledge of the original characters?
5 comments
AuthorsOpenReview ↗

Response to Reviewer RckC (1/2)

We thank the Reviewer for recognizing our "clear formulation of interaction preferences" and the "well-structured executable environment". We appreciate your questions on the validity of our evaluation and on its relation to prior work. We respond to each point below.


Point 1. How does MemPA differ from ProPerSim, UserVille, and KnowU-Bench, and what new evaluation challenges or insights does it add?

Thank you for pointing us to these three works. All three study personalized interaction behaviors, and UserVille and KnowU-Bench also give the agent an executable environment. However, none of them targets the problem MemPA addresses, and we do not see an easy way to adapt them to it.

A personal assistant's long-term interaction capability needs to be measured in an environment that is executable and where tasks stay under-specified until they are clarified over multi-turn exchanges. In such an environment the PA keeps facing interaction decisions: whether to ask, whether to act autonomously, how to handle a capability limit, and so on. Each of the three works covers part of these decisions. Evaluating a PA therefore requires a dedicated framework for interaction preferences and an interactive evaluation environment built for these decisions.

(i) UserVille studies a different task, clarification and execution by a coding agent. It defines interaction behaviors for when and how a coding agent should ask the user. This covers a subset of our preferences: Information Elicitation and part of the Expression Style dimension. Preferences specific to a personal assistant, such as Topic Management and Uncertainty Expression, are not covered. They also do not come up naturally in a coding setting.

(ii) KnowU-Bench is closer to our task. It gives the assistant an executable mobile environment. It targets content and decision preferences and touches proactive intervention lightly. Its preference space is therefore a different set and covers only proactivity in its interaction action space.

(iii) ProPerSim highlights proactivity and recommendation. It evaluates whether, when, what, and how to recommend over a long simulated horizon. The assistant cannot act, however. It can suggest ordering food, but it has no ordering tool, and no order is checked. There is also no multi-turn dialogue within a task. The complexity of real tasks, where the assistant must search the environment and clarify with the user before acting, does not arise.

MemPA differs in that its 14 attributes are chosen for this scope, and each one is constructed from cognitive science or HCI research (Appendix A.2). Based on your suggestions, we have added all three works to Related Work with the comparison below, which follows the columns of Table 1.

WorkPersonalization targetExecutable task environment (E)Preference-conditioned reactions (R)Persistent-user on-policy histories (H)Contextual isolation (I)Controlled change (C)
UserVilleInteraction preferences for clarification✓✓———
KnowU-BenchContent / decision preferences and proactive intervention✓◐◐◐—
ProPerSimRecommendation content, timing, frequency, and delivery—◐✓◐—

References

  • Kim, J., et al. (2026). ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation. ICLR 2026.
  • Sun, W., et al. (2025). Training Proactive and Personalized LLM Agents. arXiv:2511.02208.
  • Chen, T., et al. (2026). KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation. arXiv:2604.08455.

Point 2. BRS and PRS rely on LLM judges. Can they be validated against human annotations on a representative subset of probes?

Thank you for this suggestion. We agree that a human check of BRS and PRS on a subset of probes would make the metrics stronger and we have now run one.

Three annotators (AI/HCI researchers, with undergraduate degree or above) each scored 112 final probes, 28 per user and 28 per memory system, with the exact same BRS rubric and the PRS question that was given to LLM judge. A lead researcher first ran a calibration session to align their standards. Each annotator then scored independently, without seeing any model scores or the others' ratings. Each annotator agrees closely with both LLM-based scores, and less with the second judge:

Annotator 1Annotator 2Annotator 3
ComparisonρExactρExactρExact
Human BRS vs. judge BRS (Gemini)0.9583.9%0.9482.1%0.9279.5%
Human PRS vs. simulator PRS (Gemini)0.9483.9%0.9482.1%0.9378.6%
Human BRS vs. second-judge BRS (Claude Sonnet 5)0.8369.6%0.8368.8%0.8673.2%

The LLM-based scores therefore match human judgments of these subjective preferences. We have added the result to the paper.

AuthorsOpenReview ↗

Response to Reviewer RckC (2/2)

Point 3. Why only four personas from two TV series, and how representative are they of diverse interaction preferences?

Please refer to the general response.


Point 4. Ground-truth confidence varies across users (27 of 28 cells high for User A, 5 for Users C and D). Are the main findings sensitive to annotation confidence?

Good question. We wanted to start by clarifying a key distinction. The confidence levels in Table 5 measure the auto-extractor's initial certainty, not the reliability of the final ground truth. Cells below high auto-confidence are the ones the authors verified directly against the transcripts, so their final settings are unambiguous. We do include a full paragraph in Appendix B.3 explaining this. We now better signpost that section in the main text.

  1. Auto-confidence only decides which cells trigger human review. During extraction, a cell gets high confidence automatically only when a character's explicit words and their reactions to others agree across scenes. User A states outright how he wants to be helped, so 27/28 cells pass automatically. Users C and D come from The Office, where workplace exchanges are indirect and plot-driven. When signals were subtle or competing, our pipeline conservatively lowered the auto-confidence so a human would step in.

  2. Human review resolves ambiguity from transcript evidence. Two authors, who both know the preference definitions well and calibrated on them beforehand, adjudicated the cells below high auto-confidence against their cited scenes (walk-through in Appendix B.5). They either confirmed the extractor's proposal (34 Reviewed & Kept cells) or resolved competing readings of the scenes (18 of the Set by author cells). Only 7 cells had no transcript evidence and were set manually. They are in attributes such as Capability Boundary, which rarely come up in conversation between people. All the other 105 of the 112 cells are grounded in cited transcript scenes.

To show that the findings do not depend on these tiers, we recomputed PSA, CS and PET on the 87 cells where the extractor's setting was kept (53 Confirmed at high conf. + 34 Reviewed & Kept), leaving out the 25 Set by author or Specified manually cells. Values are averages over the four backbones (%).

PSACSPET
MemoryAll 11287 cellsAll 11287 cellsAll 11287 cells
File memory59.256.214.413.225.424.5
Simple RAG49.947.020.816.715.015.2
Mem050.046.822.222.914.214.2
MemOS50.047.219.922.210.810.3
No Memory49.647.021.822.211.211.3
Oracle99.099.099.5100.098.398.5

The findings hold on the subset. File memory still leads in PSA and PET, and its CS stays below No Memory. The subset keeps 12 of 18 CS pairs and 17 of 20 PET items, so the CS values move more than the others.


Point 5. Could models re-identify the original characters from the interaction histories and exploit pretrained knowledge of them?

Good question. We have two additional pieces of evidence beyond shortcut check and both show no sign that models exploit pretrained knowledge of the characters:

(i) The main results (Table 2). PSA stays between 49.6% and 50.0% for No Memory and for the three retrieval conditions. If the PA recognized the characters, we would expect a gain in these conditions, and we do not see one.

(ii) The preference change mechanism also helps to verify this. Each user has five event-driven changes that move a cell away from the setting extracted from the source character. If contamination had happened, the model would do better on the pre-event probes and then drop on the post-event probes. We do not see this. On the changed cells, accuracy after the change is about the same as before it, and under No Memory it rises from 42.5% to 52.1%.


We hope these responses address your concerns! If they do, we would appreciate it if you would consider updating your score, and we are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Revised Manuscript

Dear Reviewer RcKC,

We have uploaded a revised manuscript, with all changes marked in blue. Your comments motivated several additions that we believe strengthen our work:

  • Related work. Appendix J now discusses UserVille, KnowU-Bench, and ProPerSim and compares them along the columns of Table 1 (Table R4). The caption of Table 1 points to it.
  • Human validation of BRS and PRS. We have included the human validation in Section 4 (Simulator and Judges) , and Appendix H reports in details the agreement of three annotators with both LLM-based scores (Table R3).
  • Confidence levels of the ground truth. Section 3.2 now signpost that the confidence level only decides which cells the authors review, and points to Appendix B.3.
  • Coverage of the four users (general response). Section 3.2 now states how the four users cover the space of interaction settings, Appendix B.1 gives their personality profiles (Table R1), and Appendix C.7 describes how new users are added.

To make reading easier, new tables are temporarily numbered as R1–R4, so all other table numbers stay as cited in our responses. Appendix F.4, cited in the general response, is now Appendix F.3.

As the discussion phase is approaching its end, we are happy to provide any further details. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful review of our paper and for raising the shared concern on the number and representativeness of the four users. We answer it here in one place.

The core question that we understand is whether our findings hold for other users too. We read two parts in it: 1) whether the four users cover the range of interaction preferences a PA has to handle, and 2) whether four users provide enough evidence to support our conclusions.

  1. Coverage of interaction preferences. A math benchmark tests coverage of problem types, not every student who might solve them. Likewise, MemPA is designed to span the space of interaction settings and context shifts rather than sample a population. Across the four users, the benchmark covers 30 of the 32 discrete settings, with two users disagreeing on 60% of cells on average (16.7/28) and 18 of 56 user–attribute pairs shifting between work and personal contexts. Beyond demographic profiles, this diversity is grounded in four complementary personality and cognitive angles, which directly shape each user's interaction preference combinations:
UserOpennessConscientiousnessExtraversionAgreeablenessNeuroticismDominant Jung functions
Alowhighvery lowvery lowmidTi / Si
Bmidvery lowvery highhighmidFi / Se
Chighlowvery highmidhighNe / Te
DmidmidhighhighlowNi / Fe

Across the four users, every Big Five trait hits both low and high extremes and all eight Jungian functions appear. This gives us a compact yet comprehensive testbed that spans the preference space and is sufficient to expose where retrieval-based memory systematically fails.

  1. Evidence reliability. In MemPA, we frame each user as an entire evaluation domain (a 4-month storyline, file tree, and 99 on-policy sessions) rather than a single sample. The 28 (attribute × context) cells inside each domain are the tasks. Our design therefore evaluates 4 domains × 28 tasks, not just 4 isolated data points. Our core findings hold independently within every single domain: across all four users, File memory consistently outperforms No Memory (while exhibiting context carry-over), whereas Mem0, MemOS, and Simple RAG remain stuck at the No Memory level (Tables 2, 15–16; Appendix F.4). We argue that when a retrieval failure repeats across four distinct domains, it exposes a systemic mechanism gap (Finding 2) rather than a quirk of the particular users we chose, and detecting it does not require a large user population.

Finally, we wish to highligh that, beyond the four released domains, MemPA is built as a scalable environment with a rich interaction action space that anyone can readily extend to custom user populations from existing scripts (sitcoms are just one source; broader literature and transcripts can similarly be adapted for grounded roleplay). Our construction pipeline (transcript extraction → anonymization → world and 99-session generation, spanning both accumulation and testing) is ~95% automated, and we will release it with the code. Thus, we believe MemPA should serve the community's shared interest in having a genuinely user-centric, extensible environment for improving long-horizon human-agent interaction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.