acceptodds
Under review as a conference paper at ICLR 2027

Beyond Memorization: Benchmarking Memory Utilization in Conversational LLM Agents

18 views

Abstract

Large Language Models (LLMs) agents increasingly rely on persistent cross-session memory to support long-horizon and personalized tasks. To understand how memory is actually needed in real-world use, we annotated WildChat and identified 21,382 memory-dependent queries, of which 83.7% do not ask about memory content directly but invoke it while solving downstream tasks. Existing memory benchmarks almost exclusively evaluate the former: whether memory can be accurately recalled, leaving the latter untested. Drawing on Bloom's taxonomy, we identify three higher-order memory abilities beyond basic recall: Comprehension of memory revealed in dialogue, Apply of memory to downstream tasks, and Evaluation of whether memory should be invoked or used to correct erroneous queries. We introduce MUSE-Bench, which operationalizes these abilities into six utilization scenarios under a need-first construction pipeline. Built upon 100 carefully constructed user profiles spanning 12 life domains, MUSE-Bench contains 3,840 queries across six evaluation tasks. Notably, the Evaluation level introduces two opposing failure modes: Interference Memory, where irrelevant or misleading memory must be suppressed; and Counterfactual Query, where memory must be invoked to correct factual errors in the query, targeting the deployment risks of memory overuse and sycophancy. We evaluate 14 frontier LLMs and 20 memory baselines on MUSE-Bench, and identify two key findings: i) Frontier LLMs perform reasonably at Comprehension but show substantial gaps at Apply and Evaluation level; ii) Existing memory techniques offer limited improvement in memory utilization over conventional RAG methods. These results suggest that the next frontier of agent memory is not memorizing more, but using more wisely.

Primary area:
datasets and benchmarks
Keywords:
Long-Term Memory, Question Answer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ibSiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MUSE-Bench, a benchmark for evaluating how conversational LLM agents utilize long-term memory beyond factual recall. Motivated by an analysis of WildChat showing that 83.7% of the identified memory-dependent queries use historical information as a premise for downstream tasks rather than directly requesting its content, the authors propose a Bloom-inspired evaluation framework comprising three higher-order abilities: Understand, Apply, and Evaluate. These abilities are operationalized into six scenarios covering personalized and experiential memory understanding, user- and assistant-related memory application, interference memory suppression, and counterfactual query correction. The benchmark contains 3,840 multiple-choice questions grounded in 100 synthetic user personas across 12 life domains, with aligned conversation histories at 128K- and 1M-token scales constructed through a need-first pipeline. The authors evaluate 14 frontier LLMs and 20 memory baselines, finding that models perform well on personalized memory understanding but struggle substantially with user-related memory application and counterfactual query detection. Their experiments also suggest that existing memory methods provide limited improvements over conventional RAG baselines. Additional analyses of retrieval quality, context length, memory-free baselines, and recall-versus-utilization performance highlight both retrieval failures and difficulties in recognizing when necessary memory is missing.

Critical strengths
  • The paper provides extensive experiments covering 14 frontier LLMs and 20 memory baselines, complemented by retrieval-quality analyses, long-context evaluations at 128K and 1M tokens, ablation studies, and recall-versus-utilization comparisons. These experiments provide valuable insights into the limitations of existing models and memory systems.
  • The paper presents a well-structured evaluation framework covering six memory utilization scenarios across three cognitive levels. The need-first construction pipeline, together with cross-model verification and human validation, demonstrates considerable effort in building a benchmark for evaluating memory capabilities beyond factual recall.
Critical weaknesses
  • The novelty of this paper is limited over existing benchmarks. Although MUSE-Bench provides a unified evaluation framework, several of its core ideas and evaluation scenarios have already been explored in prior work. For example, LoCoMo-Plus and Mem2ActBench investigate implicit memory utilization beyond factual recall; OP-Bench and BenchPreS study inappropriate memory application; STALE examines stale-memory handling and resistance to incorrect premises; and UtilMem and LoCoMo-Conv address related challenges involving implicit retrieval, memory integration, or counterfactual queries. While some of these works are briefly discussed in the appendix, the paper does not sufficiently clarify what fundamentally new empirical insights its unified framework provides beyond prior findings.
  • The benchmark is entirely synthetic and relies exclusively on multiple-choice evaluation. Although the WildChat analysis supports the motivation and human evaluations assess data quality, these do not directly establish whether the synthetic scenarios accurately reflect real-world memory utilization. Moreover, selecting the correct answer among predefined options does not necessarily demonstrate that a model can independently identify, retrieve, and appropriately incorporate relevant memories into open-ended responses. Without validation on real user interactions or open-ended generation tasks, it remains unclear how reliably the reported results reflect memory utilization capabilities in practical deployments.
Questions for authors
  • The benchmark explicitly filters queries that are consistent with multiple memories to ensure a unique gold answer. While this design improves evaluation reliability, it also excludes realistic situations where multiple relevant and valid memories may support different, equally reasonable responses. For example, a user may have repeatedly expressed a preference for budget-friendly travel while also emphasizing comfort and shorter journeys. When the user requests a flight recommendation without specifying which factor matters more, either a cheaper connecting flight or a more expensive direct flight could be justified by their memory. In such cases, an appropriate assistant might present the trade-offs or seek clarification rather than commit to a single recommendation. It remains unclear how well MUSE-Bench captures this important aspect of real-world memory utilization.

Please also see the Weaknesses section for additional questions and concerns.

I would appreciate further clarification or evidence addressing these issues during the rebuttal. Convincing responses to these concerns could positively influence my overall assessment of the paper.

10 comments
AuthorsOpenReview ↗

Response to Reviewer ibSi [Part 1]

Dear Reviewer ibSi:

Thank you so much for your thoughtful review and valuable feedback. We deeply appreciate the time and effort you devoted to evaluating our work. Below, we have provided detailed responses to your comments.

1. The Novelty of MUSE-Bench

Your Concern (W1): The novelty over existing benchmarks is limited. Although MUSE-Bench offers a unified evaluation framework, several of its core ideas and scenarios have been explored before: LoCoMo-Plus and Mem2ActBench study implicit memory utilization beyond factual recall; OP-Bench and BenchPreS study inappropriate memory application; STALE examines stale-memory handling and resistance to incorrect premises; UtilMem and LoCoMo-Conv address implicit retrieval, memory integration, or counterfactual queries. While some of these works are briefly discussed in the appendix, the paper does not sufficiently clarify what fundamentally new empirical insights the unified framework provides beyond prior findings.

Thanks for this valuable comment and for pointing us to these works. Below, we clarify from three aspects—the construction method, the coverage of memory utilization, and the scale—how MUSE-Bench differs from and goes beyond these benchmarks.

(1) Novelty of the Construction Method.

The construction of MUSE-Bench differs fundamentally from that of existing memory benchmarks, which fall into two groups. The first builds on an existing dialogue corpus, most often LoCoMo, and rewrites its questions or generates new ones from its content (e.g., LoCoMo-Plus, OP-Bench, LoCoMo-Conv). The second first synthesizes memories or dialogue histories and then derives queries from them (e.g., LoCoMo, Mem2ActBench, STALE, UtilMem). Both follow a memory-first order: the memory is fixed first, and the query is then written to target it. This order imposes two limitations:

  • Limited query scenarios. Every query must be answerable from memory that already exists, so the space of queries narrows to what the stored content can support, such as asking back a fact or choosing a response consistent with a recorded preference. Queries are shaped by what the memory contains rather than by the situations in which users actually need it.
  • Limited forms of utilization. Since the answer is bounded by the content of the memory, higher-order utilization, in which memory is combined with new circumstances and world knowledge, does not arise naturally. Consider a user who once mentioned a 30-minute commute and now says their workday starts at 9 a.m. Answering when to leave requires the stored commute, the new start time, and the common knowledge to combine them; none of these alone gives the answer. A generator that starts from the commute memory has no reason to introduce the schedule change; such a query arises naturally only when one starts from the user's need.

MUSE-Bench inverts this order. We begin from a persona, identify a need the user would realistically bring to an assistant, and only then derive the memories this need requires and the dialogue history that carries them (Section 2.4). The query therefore reflects a real situation rather than the content of a memory, and the memory becomes one input among several, together with the new circumstances stated in the query, other memories, and world knowledge. We have revised Section 2.4 (Page 5, highlighted in blue) to state the two corresponding advantages of need-first construction explicitly.

(2) Novelty in Memory Utilization.

Most of the benchmarks raised by the reviewer evaluate a specific aspect of memory utilization, such as recognizing implicit memory cues (e.g., LoCoMo-Plus) or suppressing inappropriate memory (e.g., OP-Bench). MUSE-Bench instead evaluates memory utilization as a whole, across three levels (Understand, Apply, and Evaluate), so that results across levels are directly comparable.

This holistic design yields a finding that no single-aspect benchmark can produce: trusting the memory and trusting the query pull in opposite directions within the same model. Interference and Counterfactual accuracy are moderately negatively correlated across models (Spearman ρ = −0.53 over the 14 LLMs at 128K, and −0.66 over the 8 LLMs evaluated at 1M), and no model performs well on both. At 1M tokens, for instance, Claude Sonnet 4.6 reaches 0.862 on Interference but only 0.057 on Counterfactual. Models that resist misleading memory tend to accept false premises, and vice versa; such a trade-off can only be observed when both abilities are evaluated on the same models, and remains invisible to benchmarks that evaluate only one of them.

AuthorsOpenReview ↗

Response to Reviewer ibSi [Part 2]

(2) Novelty in Memory Utilization. (Supplement)

Although several of these benchmarks use names similar to ours (e.g., implicit memory, memory utilization, counterfactual or false-premise queries), they evaluate fundamentally different abilities. The key differences are as follows:

  • Mem2ActBench is the benchmark that differs most from MUSE-Bench. It evaluates memory in tool-use tasks, where remembered information is used to ground the parameters of tool calls, whereas MUSE-Bench evaluates how memory is used in conversational question answering.
  • LoCoMo-Plus evaluates whether a semantically distant utterance can trigger an earlier implicit cue, such as a user's state, goal, or value: a response counts as correct as long as it acknowledges the cue, regardless of whether the cue is used correctly. It is therefore in essence a memory-triggering problem. In MUSE-Bench, recalling the relevant memory is only a necessary condition; the answer is correct only if the memory is used correctly for the request.
  • OP-Bench evaluates over-personalization, i.e., the side effects of misusing memory, such as injecting irrelevant personal details, repeating the same memories, or sycophantically agreeing with the user. Its queries do not require memory to be answered well (memory-free agents in fact score best on it), so it examines only whether memory degrades the response. In MUSE-Bench, memory is a necessary premise for completing the task, and we evaluate both when memory should be used and when it should not.
  • BenchPreS evaluates whether stored user preferences should be applied or suppressed in specific communication settings, such as writing a formal letter to a third party. Its memory is a user profile given directly in the prompt, without any dialogue history. MUSE-Bench evaluates memory utilization over long dialogue histories, and covers not only preferences but also factual memories and conflicts between memory and the query.
  • STALE also includes categories similar to ours, such as Premise Resistance, which tests whether a model rejects queries that presuppose an outdated state. However, all of its probes stem from the same conflict, an earlier memory invalidated by later information, and even its false premises are simply the outdated state. It is thus in essence a single memory-updating problem, whereas MUSE-Bench evaluates memory utilization as a whole, from understanding memory and applying it to new situations to judging whether it should be used at all.
  • UtilMem (concurrent) uses the term memory utilization in a narrower sense: its queries ask models to organize, list, or integrate evidence scattered across sessions into summaries or plans, and answers that go beyond what the conversations state are penalized as ungrounded inference. Its utilization thus amounts to reorganizing recorded content. In MUSE-Bench, the answer is stated in no memory: the model must combine memory with the new circumstances in the query and with world knowledge to infer it, as in the commute example above.
  • LoCoMo-Conv (concurrent) rewrites LoCoMo questions into conversational forms, so the information being asked about remains LoCoMo's original facts; it is essentially fact recall in a conversational wrapper. Moreover, while it includes counterfactual queries, it does not evaluate interference, i.e., cases where using memory would mislead the response. With only one side evaluated, the trade-off between trusting the memory and trusting the query described above cannot be observed.

In addition, none of these benchmarks covers experience memory, i.e., requirements imposed by external constraints rather than by the user's own preferences, and none evaluates the use of the assistant's own earlier outputs as memory.

We have added a discussion of these benchmarks in Appendix F Related Work, §F.1 Memory Benchmark (Page 39), and accordingly revised the claim in Section 1 Introduction (Page 2) that Apply and Evaluate "remain systematically unevaluated" to state that existing benchmarks touching them each target a single aspect; both changes are highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer ibSi [Part 3]

(3) Scale.

Beyond the motivation for its construction, the scale of a benchmark's data and evaluation is an integral part of its contribution. Evaluation on a small number of scenarios, queries, or memories can hardly reflect the problems that arise in real use, and conclusions generalize only when they hold across a broad range of frontier models and memory baselines. In these respects, several of the listed benchmarks are limited in the number of queries, the length of memory context, or the range of evaluated systems, as summarized below.

Benchmark#QueriesMemory ContextQuery Type#Frontier LLMs#Memory Methods
Mem2ActBench400~3KObj.07
LoCoMo-Plus401~20KSubj.54
OP-Bench1,700LoCoMo (~20K)Subj.45
BenchPreS390N/ASubj.7–
STALE1,200≤150KObj.65
UtilMem1,71780K–220KObj.26
LoCoMo-Conv†6,581LoCoMo (~20K)Obj.05
MUSE-Bench3,840128K / 1MObj. + Subj.1421

#Frontier LLMs counts the closed-source API models and open-weight flagship models with over 100B parameters that are evaluated as the answering model; #Memory Methods counts distinct memory systems and retrieval baselines, with variants of the same system counted once.

†Although LoCoMo-Conv contains more queries than MUSE-Bench, they consist of 1,986 dialog, 1,986 implicit, and 1,540 counterfactual rewrites of the same LoCoMo question set, plus 1,069 compositions of question pairs from it, so their content and the information they ask about largely overlap. Its context is also short and limited to LoCoMo's ten conversations.

Reviewer_jMzeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MUSE-Bench evaluates how conversational agents use memory during tasks, beyond simply recalling it. It contains 3,840 questions based on 100 user profiles, organized into six scenarios covering comprehension, application, and evaluation. These include deciding when to ignore irrelevant memory and when to flag a conflict with the user’s current query. Experiments with 14 LLMs and 20 memory baselines show uneven performance across scenarios: models that recall information accurately can still struggle to use it in their responses.

Critical strengths

MUSE-Bench evaluates how conversational agents use memory during tasks, beyond simply recalling it. It contains 3,840 questions based on 100 user profiles, organized into six scenarios covering comprehension, application, and evaluation. These include deciding when to ignore irrelevant memory and when to flag a conflict with the user’s current query. Experiments with 14 LLMs and 20 memory baselines show uneven performance across scenarios: models that recall information accurately can still struggle to use it in their responses.

Critical weaknesses

My main concern is whether the low scores come from failing to retrieve the right memory or failing to use it. Section 3.6 helps, but I would still like to see results with the correct memory supplied directly. Without that test, I do not think the appendix has enough evidence to attribute an example that all models get wrong to unconditional agreement with the user. I also find some of the conclusions too broad. The reported 33.79% is for the User subset of Apply, while the Assistant subset scores 86.59%. This difference should be reflected in the discussion of application ability. Similarly, the comparison of memory methods mainly uses one 7B backbone and fixed retrieval settings; the conclusions should be limited to those conditions.

I am also unsure how the conflict questions handle facts that may have changed. In the refrigerator example, a user who previously owned a Whirlpool may now own a GE. Asking whether they replaced it makes sense, but how do models respond when the user explicitly says they did? Paired examples would help establish whether the benchmark distinguishes accepting a valid update from overlooking a contradiction. Finally, the multiple-choice format limits what we can conclude about practical task performance. Choosing a response from a list does not show that a model can produce it on its own or carry out the task. The claim that adding “I don’t know” prevents guessing also seems too strong.

Questions for authors

As weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer jMze [Part 1]

Dear Reviewer jMze:

Thank you very much for your careful review and constructive feedback. We sincerely appreciate the time and effort you have dedicated to evaluating our work. Below, we provide our detailed responses to each of your comments.

1. Performance with Supporting Memory: Retrieval vs. Utilization

Your Concern (W1): My main concern is whether the low scores come from failing to retrieve the right memory or failing to use it. Section 3.6 helps, but I would still like to see results with the correct memory supplied directly. Without that test, I do not think the appendix has enough evidence to attribute an example that all models get wrong to unconditional agreement with the user.

Thanks for this insightful comment. To verify whether the low scores stem from failing to retrieve the right memory or from failing to use it, we add an ablation on the supporting memories (SM) of each query:

  • SM-Only: only the SM of each query is supplied in the context, i.e., the correct memory is given directly.

All other settings are kept consistent with the main experiments, and the ablation is evaluated on Qwen2.5-7B-Instruct and GPT-5.2.

Results. We report ACC on each scenario together with the overall ACC and HR.

Qwen2.5-7B-InstructOverall ACCOverall HRPersonalizedExperienceUserAssistantInterferenceCounterfactual
Full-Context0.43230.56310.87380.38760.18600.52780.42520.0428
w/o Memory0.27750.47750.40080.25600.07750.35630.79530.0592
SM-Only0.53620.40340.91400.52630.28790.69040.53940.1612
GPT-5.2Overall ACCOverall HRPersonalizedExperienceUserAssistantInterferenceCounterfactual
Full-Context0.61300.30290.94290.80570.26340.89260.61950.1802
w/o Memory0.22990.19920.42040.33330.00790.05810.87740.0104
SM-Only0.73780.18570.98900.87810.53660.95250.48430.4360

Analysis.

  • The low scores stem from both retrieval and utilization, and utilization accounts for the larger share. Supplying the correct memory directly does help: the overall ACC rises from 0.4323 to 0.5362 on Qwen2.5-7B-Instruct and from 0.6130 to 0.7378 on GPT-5.2, confirming that locating the right memory in a long history is a genuine source of difficulty. Yet perfect retrieval recovers only 10.4 and 12.5 points, while 46.4 and 26.2 points of headroom remain even after it is granted. The low scores are therefore not attributable to retrieval alone—a substantial part of the gap lies in utilizing the memory once it is available.
  • The utilization gap is most pronounced on Qwen2.5-7B-Instruct. For Qwen2.5-7B-Instruct the residual gap under perfect retrieval is 4.5 times the gain that perfect retrieval itself brings, against 2.1 times for GPT-5.2, so the smaller model is limited by utilization to a markedly greater extent. Even when the supporting memory is the only content in its context, it reaches just 0.5362 overall, with User at 0.2879 and Counterfactual at 0.1612 against Personalized at 0.9140; the same ordering holds for GPT-5.2 (User 0.5366, Counterfactual 0.4360 vs. Personalized 0.9890). Handing the model exactly the evidence it needs thus leaves it far from the performance the benchmark ultimately asks for.

We would also like to clarify how we see the relation between the two. The difficulty of memory utilization does not reside solely in the using step—whether the model answers correctly once the supporting memory is laid before it. In a recall-style query the user names what is to be recalled, so the target is handed over by the question itself. In MUSE-Bench the request never states which memory bears on it, and recognizing that some earlier session is relevant at all, and which one, is part of what utilization means in realistic use. Retrieval difficulty is therefore not a confound to be factored out of the utilization gap but a constituent of it, and the SM-Only setting is best read as an upper bound that removes one of the two difficulties rather than as the condition under which the intended ability is measured. We have added these results and the corresponding analysis to Appendix D.3 Ablation Study of Memory Context (Page 31), highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer jMze [Part 2]

2. Revision of Claims in the Paper

Your Concern (W2): I also find some of the conclusions too broad. The reported 33.79% is for the User subset of Apply, while the Assistant subset scores 86.59%. This difference should be reflected in the discussion of application ability.

Your Concern (W3): Similarly, the comparison of memory methods mainly uses one 7B backbone and fixed retrieval settings; the conclusions should be limited to those conditions.

Thanks for pointing this out, and we apologize for any overstatement in the original manuscript. We have revised the corresponding parts in the updated PDF, with all revised text highlighted in blue.

On the Apply subsets (W2). The 0.3379 accuracy is indeed specific to the User subset of Apply, while the Assistant subset reaches 0.8659. This difference is rooted in our construction. Considering that in real use, users tend to modify rather than reason over an assistant's earlier responses, we construct assistant-related needs as targeted edits to these responses (e.g., adjusting a proposed plan), whereas user-related needs require integrating remembered facts with further reasoning or world knowledge (Appendix B). We have therefore restricted all claims of an Apply-level gap to user memory application, and added this explanation for the higher accuracy of the Assistant subset. These revisions are made in Section 3.2 (Page 7), as well as in the corresponding findings of the Abstract (Page 1), Introduction (Page 3), and Conclusion (Page 10).

On the scope of the memory-method comparison (W3). In addition, we have limited the conclusions of the memory-method comparison to the Qwen2.5-7B-Instruct backbone and the fixed top-5 retrieval setting used in our experiments. These revisions are made in Section 3.3 (Page 7), as well as in the corresponding findings of the Abstract (Page 1), Introduction (Page 3), and Conclusion (Page 10).

We apologize again for these imprecise statements in the original manuscript, and thank the reviewer once more for the valuable comments.

AuthorsOpenReview ↗

Response to Reviewer jMze [Part 3]

3. Ablation on Memory Updates

Your Concern (W4): I am also unsure how the conflict questions handle facts that may have changed. In the refrigerator example, a user who previously owned a Whirlpool may now own a GE. Asking whether they replaced it makes sense, but how do models respond when the user explicitly says they did? Paired examples would help establish whether the benchmark distinguishes accepting a valid update from overlooking a contradiction.

Thanks for this valuable suggestion, and we apologize for any confusion caused. The counterfactual queries in MUSE-Bench are designed to examine one specific behavior: when the premise of a query contradicts what the memory records, the model should surface the discrepancy instead of committing to either side.

That said, we fully agree with the reviewer that this design alone cannot tell whether a model genuinely detects the discrepancy or merely clings to whatever the history records, since every counterfactual query presents a conflict the model is expected to reject. To fill this gap, we construct a matched information-update subset, where each counterfactual query is paired with a legitimate-update counterpart that differs only in whether an explicit update is present. The update is introduced at two levels:

  • Memory Update: the fact is explicitly updated in a later session (e.g., the user mentions the replacement), and the query is consistent with the updated fact. The model should follow the latest memory.
  • Query Update: the user explicitly states the update in the query itself. The model should accept it directly.

We construct matched pairs for all 383 counterfactual queries in GPT-5.2's evaluation set (304 of which fall in the Qwen2.5-7B evaluation range) and evaluate under the same full-context 128K setting as the main experiments.

Results.

Qwen2.5-7B ACCQwen2.5-7B HRGPT-5.2 ACCGPT-5.2 HR
Counterfactual (Original)0.04280.95720.18020.7911
Memory Update0.22370.76970.40470.5666
Query Update0.35860.63490.51170.4595

Analysis.

  • Performance rises substantially once an update is present, and rises further when the update is stated in the query. Relative to the original counterfactual queries, Memory Update lifts ACC from 0.0428 to 0.2237 on Qwen2.5-7B and from 0.1802 to 0.4047 on GPT-5.2, while Query Update lifts them further to 0.3586 and 0.5117. The gain is consistently larger when the update is announced in the request itself (+31.6 and +33.2 points) than when it is only recorded in a later session (+18.1 and +22.5 points). This ordering is what one would expect if the cost of handling an update lies mainly in noticing it: a cue placed in the query requires no searching, whereas one buried in the history must first be found.
  • The difficulty of counterfactual queries lies in detecting a conflict that nothing announces. Both models answer the update counterparts far more often than the originals, so they are evidently capable of revising an answer once a change is signaled. What they fail at is the case where no signal is given and the discrepancy has to be surfaced by the model on its own initiative, where ACC stays at 0.0428 and 0.1802. The low scores therefore reflect this missing initiative rather than a rigid attachment to stale history, and it is precisely this self-initiated detection that the counterfactual scenario is designed to measure.

These results confirm that the counterfactual scenario in MUSE-Bench measures the ability to detect genuine contradictions while remaining receptive to valid updates. We have added this matched subset and the corresponding analysis to Appendix D.5 Counterfactual Queries with Explicit Updates (Page 33), highlighted in blue.

Reviewer_ufwfOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MUSE-Bench evaluates how conversational agents understand, apply, and judge the appropriateness of long-term memory. It contains 3,840 multiple-choice questions across six scenarios, grounded in 100 synthetic personas, with aligned 128K- and 1M-context versions. Experiments with 14 LLMs and 20 memory methods show stronger personalized understanding than user-memory application or contradictory-premise detection. The paper includes human validation, retrieval analysis, and no-memory ablations.

Critical strengths
  • Useful evaluation scope. The six scenarios cover practical demands beyond direct recall, including when an assistant should refrain from applying memory or question a conflicting premise.
  • Broad empirical coverage. Diverse personas, aligned context lengths, numerous evaluated systems, human checks, and a no-memory ablation make the benchmark a potentially valuable resource.
Critical weaknesses
  • Failure attribution remains incomplete. Removing all history establishes memory dependence, but does not isolate locating relevant evidence from reasoning with it. Gold-memory inputs, targeted removal of necessary facts, and paired histories with changed facts would substantiate the claimed utilization gap.
  • Related-work positioning is incomplete. StrategyQA, CIKQA, defeasible NLI, and TIMEDIAL establish relevant implicit, evidence-sufficiency, contextual, and temporal reasoning problems. MemInsight (Salama et al., EMNLP 2025) also evaluates memory augmentation for downstream recommendation, QA, and summarization. Discussing these works would clarify MUSE’s distinct contribution in long-term conversational evaluation; MemInsight is additionally a relevant candidate baseline.
Questions for authors

1- How does performance change with only gold supporting memories, and with those specific memories removed from the full history? 2- When a necessary remembered fact changes while the question remains fixed, does the answer change correctly? For interference tasks, does changing an irrelevant preference leave the answer unchanged? 3- How do counterfactual examples distinguish a mistaken premise from a legitimate update? Matched explicit-update cases would help establish that models detect discrepancies without overly trusting stale history.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ufwf [Part 1]

Dear Reviewer ufwf:

Thank you so much for your thoughtful review and valuable feedback. We deeply appreciate the time and effort you devoted to evaluating our work. Below, we have provided detailed responses to your comments.

1. The Failure Attribution of the Memory Utilization Gap

Your Concern (W1, Part 1): Failure attribution remains incomplete. Removing all history establishes memory dependence, but does not isolate locating relevant evidence from reasoning with it. Gold-memory inputs, targeted removal of necessary facts, ... would substantiate the claimed utilization gap.

Your Concern (Q1): How does performance change with only gold supporting memories, and with those specific memories removed from the full history?

Thanks for this insightful comment. To verify whether the memory utilization gap stems from locating the relevant evidence or from reasoning with it, we add two ablations on the supporting memories (SM) of each query:

  • SM-Only: only the SM of each query is kept in the context.
  • w/o SM: the SM of each query is removed from the full history.

All other settings are kept consistent with the main experiments, and both ablations are evaluated on Qwen2.5-7B-Instruct and GPT-5.2.

Results. We report ACC on each scenario together with the overall ACC and HR.

Qwen2.5-7B-InstructOverall ACCOverall HRPersonalizedExperienceUserAssistantInterferenceCounterfactual
Full-Context0.43230.56310.87380.38760.18600.52780.42520.0428
w/o Memory0.27750.47750.40080.25600.07750.35630.79530.0592
SM-Only0.53620.40340.91400.52630.28790.69040.53940.1612
w/o SM0.33850.64020.57980.31340.11300.49000.59450.0329
GPT-5.2Overall ACCOverall HRPersonalizedExperienceUserAssistantInterferenceCounterfactual
Full-Context0.61300.30290.94290.80570.26340.89260.61950.1802
w/o Memory0.22990.19920.42040.33330.00790.05810.87740.0104
SM-Only0.73780.18570.98900.87810.53660.95250.48430.4360
w/o SM0.34820.44950.57190.45710.08630.31340.84590.0809

Analysis.

  • Memory utilization remains challenging even when the supporting memories are given. Under SM-Only, Qwen2.5-7B-Instruct raises the overall ACC from 0.4323 (Full-Context) to 0.5362, confirming that locating evidence in long histories is part of the difficulty. However, User (0.2879) and Counterfactual (0.1612) still lag far behind Personalized (0.9140), indicating that reasoning with the memory—not merely finding it—remains the primary bottleneck, especially for smaller models.
  • Removing only the supporting memories increases the hallucination rate beyond the no-memory baseline. Compared to w/o Memory (HR 0.4775), w/o SM raises the HR of Qwen2.5-7B-Instruct to 0.6402—the non-supporting sessions provide misleading context that induces more hallucinations than having no memory at all. This phenomenon persists even on GPT-5.2 (w/o SM HR 0.4495 vs. w/o Memory HR 0.1992).
  • These findings are consistent across models. On GPT-5.2, the SM-Only ACC on User (0.5366) and Counterfactual (0.4360) still lags behind Personalized (0.9890), and w/o SM decreases the overall ACC from 0.6130 to 0.3482, confirming the same pattern.

These results confirm that the memory utilization gap stems from reasoning with memory rather than from locating it. We have added these results and the corresponding analysis to Appendix D.3 Ablation Study of Memory Context (Page 31), highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer ufwf [Part 2]

2. The Memory-Edit Ablation of Answer Grounding

Your Concern (W1, Part 2): ... paired histories with changed facts would substantiate the claimed utilization gap.

Your Concern (Q2): When a necessary remembered fact changes while the question remains fixed, does the answer change correctly? For interference tasks, does changing an irrelevant preference leave the answer unchanged?

Thanks for this insightful question. Removing the history shows that memory is necessary, but not that a correct answer is derived from the specific content of that memory, since a model may still reach the right option through persona priors or because the option looks the most plausible on its own. To test this, we build paired inputs that edit only the supporting memory and keep the query and all other sessions fixed:

  • Fact Change: for Experience, User, Assistant, and Counterfactual queries, we use GPT-5.2 to rewrite the relevant turns of the supporting sessions so that the fact the answer depends on takes a different value. The original correct option is kept verbatim as a distractor, and another option becomes the new correct answer.
  • Irrelevant Preference Change: for Interference queries, whose correct answer should not depend on the user's irrelevant preference, we replace that preference with a different but equally tempting one and rewrite the distractors to reflect it. The correct option stays unchanged.

Personalized queries are left unedited. Each edited item is checked automatically and then cross-validated by two LLMs, and only items on which both validators agree with the expected answer are kept. Only these retained items are re-evaluated with the edited memory; all other queries, including all Personalized queries, keep their original inputs and predictions. All other settings follow the main experiments.

Results.

Overall ACCOverall HRPersonalizedExperienceUserAssistantCounterfactualInterference†
Qwen2.5-7B Original0.43230.56310.87380.38760.18600.52780.04280.4252
Qwen2.5-7B Changed0.39230.60310.87380.30620.13510.46100.00330.3937
GPT-5.2 Original0.61300.30290.94290.80570.26340.89260.18020.6195
GPT-5.2 Changed0.60960.30570.94290.82670.23350.91200.17750.6132

†The Interference column reports Irrelevant Preference Change. Experience, User, Assistant, Counterfactual, and Overall are computed under Fact Change, and Personalized queries are not edited. Scenario columns report ACC.

Analysis.

  • When a relevant fact changes, GPT-5.2 changes its answer accordingly. After Fact Change, the original correct option is kept verbatim as a distractor, so a model answering from priors rather than memory would keep choosing it and lose the edited items it originally answered correctly. GPT-5.2 instead retains nearly all of its accuracy on the edited items, i.e., it moves to the new answer, showing that its correct answers are derived from the content of the memory.
  • The smaller model is only partly grounded in memory. Qwen2.5-7B shows a clearly larger drop on the edited items, concentrated on User, Experience, and Assistant, suggesting that part of its original correct answers came from priors rather than memory and thus do not survive the change.
  • When only an irrelevant preference changes, the answers largely stay the same. On Interference, GPT-5.2 is almost unaffected and Qwen2.5-7B shows only a modest drop, so the answers are robust to memory that should not matter.

These results confirm that, for capable models, answers on MUSE-Bench track the specific content of the memory rather than content-independent shortcuts, while the weaker grounding of the smaller model reflects the same memory utilization gap that MUSE-Bench is designed to expose. We have added these results and the corresponding analysis to Appendix D.4 Memory-Edit Ablation of Answer Grounding (Page 32), highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer ufwf [Part 3]

3. The Memory Update Ablation of Counterfactual Query

Your Concern (Q3): How do counterfactual examples distinguish a mistaken premise from a legitimate update? Matched explicit-update cases would help establish that models detect discrepancies without overly trusting stale history.

Thanks for this valuable question, and we apologize for any confusion caused. The counterfactual queries in MUSE-Bench are mainly intended to examine whether, when the query conflicts with the memory, the model can point out the discrepancy rather than blindly following either side.

That said, we fully agree with the reviewer that, without matched update cases, it remains unclear whether a model truly detects the discrepancy or simply over-trusts the stale history. To fill this gap, we construct a matched information-update subset, where each counterfactual query is paired with a legitimate-update counterpart that differs only in whether an explicit update is present. The update is introduced at two levels:

  • Memory Update: the original fact is explicitly updated in a later session, and the query is consistent with the updated fact. The model should follow the latest memory rather than flag a discrepancy.
  • Query Update: the user explicitly states the update in the query. The model should accept the update rather than treat it as a mistaken premise.

In this way, the expected behavior of a counterfactual query and its update counterparts differs only because of the update itself. This allows us to directly examine whether models flag discrepancies for the right reason. We construct matched pairs for all 383 counterfactual queries in GPT-5.2's evaluation set (304 of which fall in the Qwen2.5-7B evaluation range), and all other settings are kept consistent with the main experiments.

Results. We report ACC and HR on the matched counterfactual subset.

Qwen2.5-7B-InstructACCHR
Counterfactual (Original)0.04280.9572
Memory Update0.22370.7697
Query Update0.35860.6349
GPT-5.2ACCHR
Counterfactual (Original)0.18020.7911
Memory Update0.40470.5666
Query Update0.51170.4595

Analysis.

  • The information-update subset effectively separates mistaken premises from legitimate updates. When the same fact is accompanied by an explicit update (Memory Update or Query Update), both models show substantially higher ACC—Qwen2.5-7B rises from 0.0428 to 0.2237 (Memory) and 0.3586 (Query); GPT-5.2 rises from 0.1802 to 0.4047 and 0.5117, respectively. This confirms that models can distinguish between a stale fact and a legitimately updated one when the update cue is present.
  • The difficulty of counterfactual queries lies in detecting discrepancies, not in overly trusting stale history. On the original counterfactual queries (where memory has not been updated), both models achieve very low ACC (Qwen 0.0428, GPT-5.2 0.1802), primarily because they fail to identify the discrepancy. Yet once the same fact receives an explicit update, ACC increases substantially, indicating that models do not blindly trust stale history—they can follow updates when provided.

These results confirm that the counterfactual queries in MUSE-Bench measure the ability to detect mistaken premises, rather than rewarding models for blindly sticking to stale history. We have added this matched subset and the corresponding analysis to Appendix D.5 Counterfactual Queries with Explicit Updates (Page 33), highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer ufwf [Part 4]

4. Supplementary Related Work

Your Concern (W2, Part 1): Related-work positioning is incomplete. StrategyQA, CIKQA, defeasible NLI, and TIMEDIAL establish relevant implicit, evidence-sufficiency, contextual, and temporal reasoning problems. ... Discussing these works would clarify MUSE's distinct contribution in long-term conversational evaluation; ...

Thanks for this valuable suggestion. We have added a discussion of these four benchmarks in Appendix F Related Work, §F.1 Memory Benchmark (Page 39, highlighted in blue). Below we briefly summarize the positioning.

These four benchmarks belong to a line of reasoning evaluation outside the memory literature, and each isolates a single reasoning phenomenon by fixing both the material a model reasons over and the judgment it must return. StrategyQA poses boolean open-domain questions whose reasoning steps are left implicit, so the model must plan its own decomposition, but the target is fixed by the question and what is scored is the verdict it asks for. CIKQA pairs a commonsense question with a supplied knowledge snippet and asks whether that snippet is sufficient to settle the answer, making the provided knowledge the object under examination rather than something that must be located. Defeasible NLI hands the model a premise-hypothesis pair together with an additional sentence and asks whether that sentence strengthens or weakens the inference, so the candidate modifier is given explicitly and the required output is a judgment on it. TIMEDIAL is a cloze task over short dialogues that tests temporal commonsense, where the model selects among temporal expressions using cues confined to the few turns shown. Across all four settings, the evidence required to answer is either public world knowledge or is handed to the model as part of the input, whereas in MUSE-Bench it is specific to an individual user, buried in a long multi-session history, and never pointed to by the query. The more fundamental difference is what the model is asked to do with that evidence: these benchmarks ask for a verdict about the supplied material itself, whereas MUSE-Bench asks the model to act on what it remembers, carrying a recorded fact into a request that the record was never created for. Because the format never states which record applies, deciding whether to rely on memory at all, which record to act on, when to suppress an accurate preference that does not transfer to the present setting, and when to contradict a false premise carried by the query are themselves the abilities MUSE-Bench measures.

5. Supplementary Baseline: MemInsight

Your Concern (W2, Part 2): ... MemInsight (Salama et al., EMNLP 2025) also evaluates memory augmentation for downstream recommendation, QA, and summarization. ... MemInsight is additionally a relevant candidate baseline.

Thanks for pointing us to this work. We have evaluated MemInsight as a baseline on MUSE-Bench and added it to the main results in Table 3 (Page 8) and the per-domain breakdown in Figure 11 (Page 30). The method is described in Appendix C.1 Baselines Details (Page 28), and a discussion of MemInsight is added to Appendix F.2 RAG-based Memory Techniques (Page 40). All added content is highlighted in blue.

Results. MemInsight is evaluated under the same setting as the other memory methods in the main experiments. We report ACC on each scenario together with the overall ACC and HR, alongside NaiveRAG and LightMem for reference.

Overall ACCOverall HRPersonalizedExperienceUserAssistantInterferenceCounterfactual
NaiveRAG0.47980.51070.89880.44020.22370.60360.58660.0296
LightMem0.47790.50610.89880.44020.22590.56790.61020.0362
MemInsight0.48210.50570.85020.38520.25910.62140.68500.0296

MemInsight behaves like the other training-free memory methods on MUSE-Bench. Its overall ACC of 0.4821 sits within half a point of both NaiveRAG (0.4798) and LightMem (0.4779). Adding it therefore does not change the conclusion of Section 3.3 that, under the 7B backbone and fixed top-5 retrieval, training-free memory methods improve only marginally over RAG-based baselines.

Thanks again for the time and effort you have invested!

We hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.

Respectfully and sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.