- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MUSE-Bench, a benchmark for evaluating how conversational LLM agents utilize long-term memory beyond factual recall. Motivated by an analysis of WildChat showing that 83.7% of the identified memory-dependent queries use historical information as a premise for downstream tasks rather than directly requesting its content, the authors propose a Bloom-inspired evaluation framework comprising three higher-order abilities: Understand, Apply, and Evaluate. These abilities are operationalized into six scenarios covering personalized and experiential memory understanding, user- and assistant-related memory application, interference memory suppression, and counterfactual query correction. The benchmark contains 3,840 multiple-choice questions grounded in 100 synthetic user personas across 12 life domains, with aligned conversation histories at 128K- and 1M-token scales constructed through a need-first pipeline. The authors evaluate 14 frontier LLMs and 20 memory baselines, finding that models perform well on personalized memory understanding but struggle substantially with user-related memory application and counterfactual query detection. Their experiments also suggest that existing memory methods provide limited improvements over conventional RAG baselines. Additional analyses of retrieval quality, context length, memory-free baselines, and recall-versus-utilization performance highlight both retrieval failures and difficulties in recognizing when necessary memory is missing.
Critical strengths
- The paper provides extensive experiments covering 14 frontier LLMs and 20 memory baselines, complemented by retrieval-quality analyses, long-context evaluations at 128K and 1M tokens, ablation studies, and recall-versus-utilization comparisons. These experiments provide valuable insights into the limitations of existing models and memory systems.
- The paper presents a well-structured evaluation framework covering six memory utilization scenarios across three cognitive levels. The need-first construction pipeline, together with cross-model verification and human validation, demonstrates considerable effort in building a benchmark for evaluating memory capabilities beyond factual recall.
Critical weaknesses
- The novelty of this paper is limited over existing benchmarks. Although MUSE-Bench provides a unified evaluation framework, several of its core ideas and evaluation scenarios have already been explored in prior work. For example, LoCoMo-Plus and Mem2ActBench investigate implicit memory utilization beyond factual recall; OP-Bench and BenchPreS study inappropriate memory application; STALE examines stale-memory handling and resistance to incorrect premises; and UtilMem and LoCoMo-Conv address related challenges involving implicit retrieval, memory integration, or counterfactual queries. While some of these works are briefly discussed in the appendix, the paper does not sufficiently clarify what fundamentally new empirical insights its unified framework provides beyond prior findings.
- The benchmark is entirely synthetic and relies exclusively on multiple-choice evaluation. Although the WildChat analysis supports the motivation and human evaluations assess data quality, these do not directly establish whether the synthetic scenarios accurately reflect real-world memory utilization. Moreover, selecting the correct answer among predefined options does not necessarily demonstrate that a model can independently identify, retrieve, and appropriately incorporate relevant memories into open-ended responses. Without validation on real user interactions or open-ended generation tasks, it remains unclear how reliably the reported results reflect memory utilization capabilities in practical deployments.
Questions for authors
- The benchmark explicitly filters queries that are consistent with multiple memories to ensure a unique gold answer. While this design improves evaluation reliability, it also excludes realistic situations where multiple relevant and valid memories may support different, equally reasonable responses. For example, a user may have repeatedly expressed a preference for budget-friendly travel while also emphasizing comfort and shorter journeys. When the user requests a flight recommendation without specifying which factor matters more, either a cheaper connecting flight or a more expensive direct flight could be justified by their memory. In such cases, an appropriate assistant might present the trade-offs or seek clarification rather than commit to a single recommendation. It remains unclear how well MUSE-Bench captures this important aspect of real-world memory utilization.
Please also see the Weaknesses section for additional questions and concerns.
I would appreciate further clarification or evidence addressing these issues during the rebuttal. Convincing responses to these concerns could positively influence my overall assessment of the paper.
10 comments
Response to Reviewer ibSi [Part 2]
(2) Novelty in Memory Utilization. (Supplement)
Although several of these benchmarks use names similar to ours (e.g., implicit memory, memory utilization, counterfactual or false-premise queries), they evaluate fundamentally different abilities. The key differences are as follows:
- Mem2ActBench is the benchmark that differs most from MUSE-Bench. It evaluates memory in tool-use tasks, where remembered information is used to ground the parameters of tool calls, whereas MUSE-Bench evaluates how memory is used in conversational question answering.
- LoCoMo-Plus evaluates whether a semantically distant utterance can trigger an earlier implicit cue, such as a user's state, goal, or value: a response counts as correct as long as it acknowledges the cue, regardless of whether the cue is used correctly. It is therefore in essence a memory-triggering problem. In MUSE-Bench, recalling the relevant memory is only a necessary condition; the answer is correct only if the memory is used correctly for the request.
- OP-Bench evaluates over-personalization, i.e., the side effects of misusing memory, such as injecting irrelevant personal details, repeating the same memories, or sycophantically agreeing with the user. Its queries do not require memory to be answered well (memory-free agents in fact score best on it), so it examines only whether memory degrades the response. In MUSE-Bench, memory is a necessary premise for completing the task, and we evaluate both when memory should be used and when it should not.
- BenchPreS evaluates whether stored user preferences should be applied or suppressed in specific communication settings, such as writing a formal letter to a third party. Its memory is a user profile given directly in the prompt, without any dialogue history. MUSE-Bench evaluates memory utilization over long dialogue histories, and covers not only preferences but also factual memories and conflicts between memory and the query.
- STALE also includes categories similar to ours, such as Premise Resistance, which tests whether a model rejects queries that presuppose an outdated state. However, all of its probes stem from the same conflict, an earlier memory invalidated by later information, and even its false premises are simply the outdated state. It is thus in essence a single memory-updating problem, whereas MUSE-Bench evaluates memory utilization as a whole, from understanding memory and applying it to new situations to judging whether it should be used at all.
- UtilMem (concurrent) uses the term memory utilization in a narrower sense: its queries ask models to organize, list, or integrate evidence scattered across sessions into summaries or plans, and answers that go beyond what the conversations state are penalized as ungrounded inference. Its utilization thus amounts to reorganizing recorded content. In MUSE-Bench, the answer is stated in no memory: the model must combine memory with the new circumstances in the query and with world knowledge to infer it, as in the commute example above.
- LoCoMo-Conv (concurrent) rewrites LoCoMo questions into conversational forms, so the information being asked about remains LoCoMo's original facts; it is essentially fact recall in a conversational wrapper. Moreover, while it includes counterfactual queries, it does not evaluate interference, i.e., cases where using memory would mislead the response. With only one side evaluated, the trade-off between trusting the memory and trusting the query described above cannot be observed.
In addition, none of these benchmarks covers experience memory, i.e., requirements imposed by external constraints rather than by the user's own preferences, and none evaluates the use of the assistant's own earlier outputs as memory.
We have added a discussion of these benchmarks in Appendix F Related Work, §F.1 Memory Benchmark (Page 39), and accordingly revised the claim in Section 1 Introduction (Page 2) that Apply and Evaluate "remain systematically unevaluated" to state that existing benchmarks touching them each target a single aspect; both changes are highlighted in blue.
Response to Reviewer ibSi [Part 3]
(3) Scale.
Beyond the motivation for its construction, the scale of a benchmark's data and evaluation is an integral part of its contribution. Evaluation on a small number of scenarios, queries, or memories can hardly reflect the problems that arise in real use, and conclusions generalize only when they hold across a broad range of frontier models and memory baselines. In these respects, several of the listed benchmarks are limited in the number of queries, the length of memory context, or the range of evaluated systems, as summarized below.
| Benchmark | #Queries | Memory Context | Query Type | #Frontier LLMs | #Memory Methods |
|---|---|---|---|---|---|
| Mem2ActBench | 400 | ~3K | Obj. | 0 | 7 |
| LoCoMo-Plus | 401 | ~20K | Subj. | 5 | 4 |
| OP-Bench | 1,700 | LoCoMo (~20K) | Subj. | 4 | 5 |
| BenchPreS | 390 | N/A | Subj. | 7 | – |
| STALE | 1,200 | ≤150K | Obj. | 6 | 5 |
| UtilMem | 1,717 | 80K–220K | Obj. | 2 | 6 |
| LoCoMo-Conv† | 6,581 | LoCoMo (~20K) | Obj. | 0 | 5 |
| MUSE-Bench | 3,840 | 128K / 1M | Obj. + Subj. | 14 | 21 |
#Frontier LLMs counts the closed-source API models and open-weight flagship models with over 100B parameters that are evaluated as the answering model; #Memory Methods counts distinct memory systems and retrieval baselines, with variants of the same system counted once.
†Although LoCoMo-Conv contains more queries than MUSE-Bench, they consist of 1,986 dialog, 1,986 implicit, and 1,540 counterfactual rewrites of the same LoCoMo question set, plus 1,069 compositions of question pairs from it, so their content and the information they ask about largely overlap. Its context is also short and limited to LoCoMo's ten conversations.
Response to Reviewer ibSi [Part 1]
Dear Reviewer ibSi:
Thank you so much for your thoughtful review and valuable feedback. We deeply appreciate the time and effort you devoted to evaluating our work. Below, we have provided detailed responses to your comments.
1. The Novelty of MUSE-Bench
Thanks for this valuable comment and for pointing us to these works. Below, we clarify from three aspects—the construction method, the coverage of memory utilization, and the scale—how MUSE-Bench differs from and goes beyond these benchmarks.
(1) Novelty of the Construction Method.
The construction of MUSE-Bench differs fundamentally from that of existing memory benchmarks, which fall into two groups. The first builds on an existing dialogue corpus, most often LoCoMo, and rewrites its questions or generates new ones from its content (e.g., LoCoMo-Plus, OP-Bench, LoCoMo-Conv). The second first synthesizes memories or dialogue histories and then derives queries from them (e.g., LoCoMo, Mem2ActBench, STALE, UtilMem). Both follow a memory-first order: the memory is fixed first, and the query is then written to target it. This order imposes two limitations:
MUSE-Bench inverts this order. We begin from a persona, identify a need the user would realistically bring to an assistant, and only then derive the memories this need requires and the dialogue history that carries them (
Section 2.4). The query therefore reflects a real situation rather than the content of a memory, and the memory becomes one input among several, together with the new circumstances stated in the query, other memories, and world knowledge. We have revisedSection 2.4(Page 5, highlighted in blue) to state the two corresponding advantages of need-first construction explicitly.(2) Novelty in Memory Utilization.
Most of the benchmarks raised by the reviewer evaluate a specific aspect of memory utilization, such as recognizing implicit memory cues (e.g., LoCoMo-Plus) or suppressing inappropriate memory (e.g., OP-Bench). MUSE-Bench instead evaluates memory utilization as a whole, across three levels (Understand, Apply, and Evaluate), so that results across levels are directly comparable.
This holistic design yields a finding that no single-aspect benchmark can produce: trusting the memory and trusting the query pull in opposite directions within the same model. Interference and Counterfactual accuracy are moderately negatively correlated across models (Spearman ρ = −0.53 over the 14 LLMs at 128K, and −0.66 over the 8 LLMs evaluated at 1M), and no model performs well on both. At 1M tokens, for instance, Claude Sonnet 4.6 reaches 0.862 on Interference but only 0.057 on Counterfactual. Models that resist misleading memory tend to accept false premises, and vice versa; such a trade-off can only be observed when both abilities are evaluated on the same models, and remains invisible to benchmarks that evaluate only one of them.