- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper evaluates long-term memory across multiple users and devices through three tasks covering memory operation decisions, factual memory recognition, and contextual memory application. It also introduces LILIA-Mem, which combines relation-constrained revision with applicability-aware retrieval to improve contextual application. The evaluation shows that factual recognition alone does not fully capture a system’s ability to use memory in context.
Critical strengths
The paper provides a detailed evaluation of memory across users, devices, and evolving histories, with particular attention to Contextual Memory Application. This focus is relevant to persistent assistants that must use current facts while respecting user access rights.
Critical weaknesses
A minor concern is that existing benchmarks already test some of these dimensions separately. LILIA combines them in one benchmark, but the paper could explain more clearly what this combined evaluation adds.
Questions for authors
How well do the task scenarios and their distribution reflect real user needs? How does the computational cost of LILIA-Mem compare with baselines using the same model?
8 comments
Reviewer Comment R3-1
A minor concern is that existing benchmarks already test some of these dimensions separately. LILIA combines them in one benchmark, but the paper could explain more clearly what this combined evaluation adds.
Response
We sincerely thank the reviewer for recognizing the value of our work and experimental design. LILIA places multiple users, devices, state updates, and access control within the same continuously evolving interaction history. The goal is not merely to aggregate separate evaluation tasks, but to examine how different memory capabilities interact, particularly the gap between fact recognition and contextual application, and failures arising from the interaction of state updates and multiple usage constraints.
First, evaluating memory operations (MOD), fact recognition (FR), and contextual memory application (CMA) against the same reference-state trajectory reveals the gap between recognizing information and using it correctly.
When factual question answering and access control are evaluated on separate benchmarks, it is difficult to determine whether an incorrect response results from failure to retain the latest fact or failure to apply the available evidence appropriately. LILIA derives three complementary tasks from the same evolving event history, demonstrating that recognizing currently valid facts does not necessarily translate into correct contextual application.
In the main experiments, GPT-5-mini achieves the highest FR accuracy of 74.20%, but its CMA accuracy is only 33.95%, with just 11.80% on Visibility (VIS). Similarly, specialized memory systems such as Mem0 and LightMem achieve FR accuracy above 72% and MOD accuracy above 45%, yet their CMA accuracy remains around 35%. These differences, observed over the same evolving histories, show that fact retrieval or recognition alone can substantially overestimate a memory system's reliability in actual interactions.
Second, joint evaluation captures cascading failures between cross-device state updates and the preservation of access restrictions.
In practice, different constraints often interact throughout the lifecycle of a memory. Consider Alice, who states a preference on her personal phone and requests that it remain private. She later updates the preference using her laptop without restating the privacy restriction. Bob subsequently asks about the preference through a shared household speaker.
A benchmark evaluating only fact updates might consider the operation successful once the new value is stored. A benchmark testing only static permissions might require the system to enforce the original access label. However, under joint evaluation, the assistant must update the preference across devices while preserving the unchanged subject attribution and privacy restrictions.
Our ablation results illustrate this interaction. Replacing relation-constrained revision with direct operation planning (RCR → Direct) reduces Update accuracy from 40.00% to 27.00%, while downstream VIS accuracy decreases from 47.00% to 24.00%. Unconstrained updates may discard existing access restrictions when modifying factual values, subsequently causing unauthorized disclosure on shared devices. Joint evaluation makes it possible to identify such failures spanning memory updates and access control.
Third, evaluating cross-device continuity and multi-user isolation within the same shared history prevents systems from relying on shortcuts that work for only one task.
In histories where 50% of devices are shared by multiple users and 926 questions involve cross-device queries, the assistant must satisfy several potentially competing requirements:
- Cross-device continuity (CD): Personal facts without device restrictions must remain accessible when the same user switches devices.
- Visibility (VIS): Private information must remain protected when different users access a shared device, while legitimately shared information must remain available. VIS includes 240 authorized-disclosure and 260 nondisclosure cases.
- Subject grounding and temporal validity (SG, UI): The system must distinguish different users' valid values for the same attribute and avoid relying on outdated information.
Evaluating these requirements against the same memory state prevents the system from succeeding through indiscriminate isolation by device or current speaker, which would undermine cross-device continuity and authorized sharing. Likewise, retrieving all highly similar historical text without considering subject identity, validity, and permissions risks subject confusion, interference from outdated facts, and privacy violations.
The contribution of joint evaluation is therefore the ability to examine these interdependent requirements within a consistent evolving memory state, rather than assessing each capability in isolation.
Reviewer Comment R3-2
How well do the task scenarios and their distribution reflect real user needs?
Response (Part A: Real-World Relevance of Task Scenarios)
We thank the reviewer for this important question. We address the relationship between LILIA and real user needs from two perspectives: the real-world relevance of the task scenarios and the rationale behind the evaluation distribution.
First, LILIA's four core scenarios—subject attribution, cross-device continuity, state updates and invalidation, and visibility control—correspond to important practical requirements and known challenges in multi-user, multi-device assistant environments.
Existing AI assistants already support multi-user identification, cross-device interaction continuity, and persistent personalization to varying degrees. For example, Apple HomePod supports household member identification and personalized services [1], while Huawei's in-car XiaoYi provides voiceprint recognition and personalized interactions for different passengers [6]. Meta AI supports certain forms of conversation continuity between smart glasses and mobile applications [3], while Xiaomi Super XiaoAi and HONOR YOYO Memory offer different forms of cross-device memory synchronization [5,7]. Google Personal Intelligence supports personal information retrieval and reasoning across applications [2], and Doubao provides persistent preference memory and memory management capabilities [4].
Although these products differ in their implementations and supported functionality, they collectively illustrate practical demands for managing information about multiple individuals, maintaining interactions across devices, and updating persistent memories.
However, Google's official technical report identifies several recurring problems in deployed personal AI systems, including attributing family members' preferences to the wrong user, overgeneralizing existing preferences, confusing temporal relationships, missing subsequent cancellations, and ignoring user corrections [2].
Furthermore, both our analysis of anonymized real-world assistant logs and empirical research on smart speakers indicate that everyday assistant interactions are often brief and fragmented. Bentley et al. analyzed 65,499 real interactions from 88 households, finding a median command length of only four English words [8].
Our real-world interaction logs contain several representative examples:
(1) Cross-user, cross-device management of the same household event. A mother creates a reminder for her child's kindergarten activity on phone A. The father subsequently changes the schedule and adds clothing preparation requirements using phone B. The mother later cancels the activity because of the weather. The assistant must associate these interactions with the same event and maintain its latest state.
(2) Separation between the current interaction device and the device to which a preference applies. After establishing a preference for crosstalk playback in the car, a user later says on a phone, "Stop the crosstalk and switch to storytelling; play this on the car system from now on." The assistant must distinguish the device receiving the current instruction from the device targeted by the persistent preference.
(3) Converting fragmented immediate commands into conditional long-term preferences. A user first tells a smart speaker, "Play some soft music," then adds, "Lower the volume; my daughter is studying," and later says, "Always play this kind of music when supervising her studies." The assistant must infer the persistent preference and preserve its applicability conditions across these fragmented interactions.
(4) Distinguishing facts about different household members mentioned by the same speaker. For example, a wife may record information about herself and her husband taking the same allergy medication through a shared speaker and her personal phone. The assistant must correctly identify the subject of each fact despite the same speaker mentioning similar information about different people.
LILIA translates these real-world interaction requirements into traceable evaluation conditions through memory operation decisions (MOD), fact recognition (FR), and four contextual application tasks: Subject Grounding (SG), Cross-device Memory (CD), Update/Invalidation (UI), and Visibility (VIS).
The corresponding references are provided in Part B.
Overall Response to Reviewer R3
We sincerely thank the reviewer for the positive assessment of our work and for the constructive questions regarding the value of joint evaluation, the realism of task scenarios, and computational efficiency.
We provide three clarifications and additional analyses:
Value of joint evaluation. LILIA does more than combine existing evaluation dimensions. By evaluating memory operations (MOD), valid-fact recognition (FR), and contextual memory application (CMA) over the same evolving histories, it reveals performance gaps across memory stages, cascading failures between state updates and access control, and limitations of strategies optimized for individual tasks.
Real-world relevance and task distribution. LILIA's scenarios are informed by anonymized real-world assistant interactions and existing industrial applications. We distinguish the realistic characteristics of the underlying interaction histories from the deliberately balanced distribution of evaluation questions, which ensures adequate coverage of important memory capabilities and safety-sensitive failures.
Computational cost and performance trade-offs. We provide a detailed comparison of LLM calls and token consumption using GPT-5-mini across all systems. While LILIA-Mem incurs greater inference costs than conventional retrieval baselines, its computational requirements are comparable to those of generative memory systems. Module-level ablations further quantify the performance improvements associated with additional computation.
We address each question below with concrete examples, experimental results, and relevant references.