acceptodds
Under review as a conference paper at ICLR 2027

LILIA: Benchmarking Long-Term Memory for Multi-User, Multi-Device Assistants

18 views

Abstract

Existing benchmarks have advanced long-term memory evaluation, yet provide limited coverage of interactions in which users switch devices and devices are shared across users. This leaves an important question insufficiently examined: whether memory can sustain personalized service **for multiple users across personal and shared devices** while maintaining correct user attribution, up-to-date information, and appropriate access boundaries. To address this gap, we introduce **LILIA**, a benchmark for **L**ong-term memory **I**n mu**L**ti-user, mult**I**-device inter**A**ctions. We first specify who provides information, through which device, and how it is later revised or withdrawn. These events guide dialogue generation and determine reference states and test answers. LILIA comprises **16K multi-turn dialogue sessions in Chinese and English and 4K evaluation instances**. Three complementary tasks assess memory-informed Add, Update, and Delete decisions and their arguments; factual recognition through multiple-choice questions; and open-ended responses involving subject grounding, cross-device use, update and invalidation, and visibility constraints. A unified evaluation harness connects retrieval baselines and memory systems to shared answering and scoring interfaces. We further propose **LILIA-Mem**, which combines joint relation inference with constrained memory revision, preserving conditions and permissions not explicitly changed. Its retrieval process integrates access checks, budget-aware evidence selection, and semantic verification to support context-appropriate responses. The anonymous repository is available at [https://anonymous.4open.science/r/ICLR2027Submission](https://anonymous.4open.science/r/ICLR2027Submission).

Primary area:
datasets and benchmarks
Keywords:
Long-Term Memory, LLM Agents, Multi-User Memory, Multi-Device Interaction, Memory Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eF8aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates long-term memory across multiple users and devices through three tasks covering memory operation decisions, factual memory recognition, and contextual memory application. It also introduces LILIA-Mem, which combines relation-constrained revision with applicability-aware retrieval to improve contextual application. The evaluation shows that factual recognition alone does not fully capture a system’s ability to use memory in context.

Critical strengths

The paper provides a detailed evaluation of memory across users, devices, and evolving histories, with particular attention to Contextual Memory Application. This focus is relevant to persistent assistants that must use current facts while respecting user access rights.

Critical weaknesses

A minor concern is that existing benchmarks already test some of these dimensions separately. LILIA combines them in one benchmark, but the paper could explain more clearly what this combined evaluation adds.

Questions for authors

How well do the task scenarios and their distribution reflect real user needs? How does the computational cost of LILIA-Mem compare with baselines using the same model?

8 comments

Overall Response to Reviewer R3

We sincerely thank the reviewer for the positive assessment of our work and for the constructive questions regarding the value of joint evaluation, the realism of task scenarios, and computational efficiency.

We provide three clarifications and additional analyses:

  1. Value of joint evaluation. LILIA does more than combine existing evaluation dimensions. By evaluating memory operations (MOD), valid-fact recognition (FR), and contextual memory application (CMA) over the same evolving histories, it reveals performance gaps across memory stages, cascading failures between state updates and access control, and limitations of strategies optimized for individual tasks.

  2. Real-world relevance and task distribution. LILIA's scenarios are informed by anonymized real-world assistant interactions and existing industrial applications. We distinguish the realistic characteristics of the underlying interaction histories from the deliberately balanced distribution of evaluation questions, which ensures adequate coverage of important memory capabilities and safety-sensitive failures.

  3. Computational cost and performance trade-offs. We provide a detailed comparison of LLM calls and token consumption using GPT-5-mini across all systems. While LILIA-Mem incurs greater inference costs than conventional retrieval baselines, its computational requirements are comparable to those of generative memory systems. Module-level ablations further quantify the performance improvements associated with additional computation.

We address each question below with concrete examples, experimental results, and relevant references.

Reviewer Comment R3-1

A minor concern is that existing benchmarks already test some of these dimensions separately. LILIA combines them in one benchmark, but the paper could explain more clearly what this combined evaluation adds.

Response

We sincerely thank the reviewer for recognizing the value of our work and experimental design. LILIA places multiple users, devices, state updates, and access control within the same continuously evolving interaction history. The goal is not merely to aggregate separate evaluation tasks, but to examine how different memory capabilities interact, particularly the gap between fact recognition and contextual application, and failures arising from the interaction of state updates and multiple usage constraints.

First, evaluating memory operations (MOD), fact recognition (FR), and contextual memory application (CMA) against the same reference-state trajectory reveals the gap between recognizing information and using it correctly.

When factual question answering and access control are evaluated on separate benchmarks, it is difficult to determine whether an incorrect response results from failure to retain the latest fact or failure to apply the available evidence appropriately. LILIA derives three complementary tasks from the same evolving event history, demonstrating that recognizing currently valid facts does not necessarily translate into correct contextual application.

In the main experiments, GPT-5-mini achieves the highest FR accuracy of 74.20%, but its CMA accuracy is only 33.95%, with just 11.80% on Visibility (VIS). Similarly, specialized memory systems such as Mem0 and LightMem achieve FR accuracy above 72% and MOD accuracy above 45%, yet their CMA accuracy remains around 35%. These differences, observed over the same evolving histories, show that fact retrieval or recognition alone can substantially overestimate a memory system's reliability in actual interactions.

Second, joint evaluation captures cascading failures between cross-device state updates and the preservation of access restrictions.

In practice, different constraints often interact throughout the lifecycle of a memory. Consider Alice, who states a preference on her personal phone and requests that it remain private. She later updates the preference using her laptop without restating the privacy restriction. Bob subsequently asks about the preference through a shared household speaker.

A benchmark evaluating only fact updates might consider the operation successful once the new value is stored. A benchmark testing only static permissions might require the system to enforce the original access label. However, under joint evaluation, the assistant must update the preference across devices while preserving the unchanged subject attribution and privacy restrictions.

Our ablation results illustrate this interaction. Replacing relation-constrained revision with direct operation planning (RCR → Direct) reduces Update accuracy from 40.00% to 27.00%, while downstream VIS accuracy decreases from 47.00% to 24.00%. Unconstrained updates may discard existing access restrictions when modifying factual values, subsequently causing unauthorized disclosure on shared devices. Joint evaluation makes it possible to identify such failures spanning memory updates and access control.

Third, evaluating cross-device continuity and multi-user isolation within the same shared history prevents systems from relying on shortcuts that work for only one task.

In histories where 50% of devices are shared by multiple users and 926 questions involve cross-device queries, the assistant must satisfy several potentially competing requirements:

  • Cross-device continuity (CD): Personal facts without device restrictions must remain accessible when the same user switches devices.
  • Visibility (VIS): Private information must remain protected when different users access a shared device, while legitimately shared information must remain available. VIS includes 240 authorized-disclosure and 260 nondisclosure cases.
  • Subject grounding and temporal validity (SG, UI): The system must distinguish different users' valid values for the same attribute and avoid relying on outdated information.

Evaluating these requirements against the same memory state prevents the system from succeeding through indiscriminate isolation by device or current speaker, which would undermine cross-device continuity and authorized sharing. Likewise, retrieving all highly similar historical text without considering subject identity, validity, and permissions risks subject confusion, interference from outdated facts, and privacy violations.

The contribution of joint evaluation is therefore the ability to examine these interdependent requirements within a consistent evolving memory state, rather than assessing each capability in isolation.

Reviewer Comment R3-2

How well do the task scenarios and their distribution reflect real user needs?

Response (Part A: Real-World Relevance of Task Scenarios)

We thank the reviewer for this important question. We address the relationship between LILIA and real user needs from two perspectives: the real-world relevance of the task scenarios and the rationale behind the evaluation distribution.

First, LILIA's four core scenarios—subject attribution, cross-device continuity, state updates and invalidation, and visibility control—correspond to important practical requirements and known challenges in multi-user, multi-device assistant environments.

Existing AI assistants already support multi-user identification, cross-device interaction continuity, and persistent personalization to varying degrees. For example, Apple HomePod supports household member identification and personalized services [1], while Huawei's in-car XiaoYi provides voiceprint recognition and personalized interactions for different passengers [6]. Meta AI supports certain forms of conversation continuity between smart glasses and mobile applications [3], while Xiaomi Super XiaoAi and HONOR YOYO Memory offer different forms of cross-device memory synchronization [5,7]. Google Personal Intelligence supports personal information retrieval and reasoning across applications [2], and Doubao provides persistent preference memory and memory management capabilities [4].

Although these products differ in their implementations and supported functionality, they collectively illustrate practical demands for managing information about multiple individuals, maintaining interactions across devices, and updating persistent memories.

However, Google's official technical report identifies several recurring problems in deployed personal AI systems, including attributing family members' preferences to the wrong user, overgeneralizing existing preferences, confusing temporal relationships, missing subsequent cancellations, and ignoring user corrections [2].

Furthermore, both our analysis of anonymized real-world assistant logs and empirical research on smart speakers indicate that everyday assistant interactions are often brief and fragmented. Bentley et al. analyzed 65,499 real interactions from 88 households, finding a median command length of only four English words [8].

Our real-world interaction logs contain several representative examples:

(1) Cross-user, cross-device management of the same household event. A mother creates a reminder for her child's kindergarten activity on phone A. The father subsequently changes the schedule and adds clothing preparation requirements using phone B. The mother later cancels the activity because of the weather. The assistant must associate these interactions with the same event and maintain its latest state.

(2) Separation between the current interaction device and the device to which a preference applies. After establishing a preference for crosstalk playback in the car, a user later says on a phone, "Stop the crosstalk and switch to storytelling; play this on the car system from now on." The assistant must distinguish the device receiving the current instruction from the device targeted by the persistent preference.

(3) Converting fragmented immediate commands into conditional long-term preferences. A user first tells a smart speaker, "Play some soft music," then adds, "Lower the volume; my daughter is studying," and later says, "Always play this kind of music when supervising her studies." The assistant must infer the persistent preference and preserve its applicability conditions across these fragmented interactions.

(4) Distinguishing facts about different household members mentioned by the same speaker. For example, a wife may record information about herself and her husband taking the same allergy medication through a shared speaker and her personal phone. The assistant must correctly identify the subject of each fact despite the same speaker mentioning similar information about different people.

LILIA translates these real-world interaction requirements into traceable evaluation conditions through memory operation decisions (MOD), fact recognition (FR), and four contextual application tasks: Subject Grounding (SG), Cross-device Memory (CD), Update/Invalidation (UI), and Visibility (VIS).

The corresponding references are provided in Part B.

Reviewer_CkCROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LILIA, a benchmark for long-term memory in multi-user, multi-device assistants, comprising 20 environments, 16K dialogue sessions, and 4K evaluation instances in Chinese and English. Its event-grounded construction links dialogue histories to evolving reference states. Three tasks evaluate memory-operation decisions, factual recognition, and contextual application, including user attribution, cross-device continuity, updates, and visibility. The authors also propose LILIA-Mem, combining relation-constrained memory revision with applicability-aware retrieval. Experiments compare thirteen configurations and show that strong factual recognition does not necessarily imply correct contextual use.

Critical strengths
  1. The benchmark addresses a practically relevant setting where assistants must preserve user identity, temporal validity, and access restrictions alongside factual content.
  2. Event-grounded construction provides traceable reference states and explicit history boundaries, supporting reproducible evaluation.
  3. Human review covers 70% of reference annotations, and the history-support diagnostic and controlled component ablations provide useful validation.
Critical weaknesses
  1. Median environment history is approximately 43.8K tokens, yet the context-based assistants use summaries and recent dialogue rather than complete histories. Without a full-history baseline, it is difficult to distinguish memory-management failures from summarization loss or contextual reasoning difficulties. The no-history versus annotated-support comparison establishes history dependence but does not resolve this distinction.

  2. The scorer–human audit contains 2,705 correct outputs among 2,800 samples, substantially different from the approximately 40–48% overall accuracies in the main experiments. The sampling procedure and representativeness need clarification. Moreover, the judge accepts 18 of 95 human-incorrect outputs. Task- and system-specific audits are needed to establish ranking reliability.

  3. LILIA-Mem improves overall accuracy over Dense RAG by only 2.17 percentage points, without repeated-run results or confidence intervals. Evidence interfaces also differ: Mem0 and LightMem expose all available memories, while LILIA-Mem delivers tightly selected evidence. Matched-budget comparisons and standard retrieval configurations would better isolate the proposed method’s contribution.

  4. Related benchmarks already evaluate user attribution, updates, and access control, while LILIA’s cross-device questions primarily concern unchanged facts queried after switching endpoints. Device-information ablations, tasks where device conditions change the correct answer, and tests combining multiple constraints would better demonstrate the distinctive contribution of the multi-device setting.

Questions for authors

Does MOD penalize additional incorrect operations? Can the evaluation detect unnecessary memory changes during ordinary interactions, rather than only assess Add, Update, and Delete cases?

11 comments

Overall Response to Reviewer R2

We thank the reviewer for the detailed evaluation and constructive suggestions. The concerns primarily relate to experimental controls, evaluation reliability, and the validity of our conclusions. To address these concerns, we have conducted additional experiments covering full-history baselines, matched-budget comparisons, repeated runs, device-specific evaluations, and targeted ablations, while further clarifying the evaluation protocol.

Specifically:

  1. Full-history baseline: We evaluated GPT-5-mini on all 4,000 instances using complete historical dialogue prefixes without summarization. Although this improves performance over the original summary-based setting, LILIA-Mem still achieves higher overall and contextual application accuracy, particularly on Update/Invalidation (UI) and Visibility (VIS).

  2. Scoring reliability: We clarify the distinction between task accuracy and judge–human agreement. The original audit achieved 96.86% agreement between the LLM judge and human assessments, with 18 false acceptances and 70 false rejections.

  3. Fairness and stability: We conducted matched-budget comparisons using a maximum of 10 evidence records and 4,096 tokens, enabled native retrieval for Mem0 and LightMem, and performed three independent runs. LILIA-Mem maintains its advantages in overall accuracy and contextual memory application under these controlled conditions.

  4. Multi-device contribution: We extended the evaluation with 300 targeted questions covering cross-device continuity, compound constraints, and device-dependent answers. Removing structured device metadata substantially reduces performance on device-dependent tasks, confirming the importance of explicit device context.

  5. MOD scoring: We clarify that MOD requires an exact match of the target operation and its parameters but does not separately penalize additional operation declarations, and explain the rationale for this design.

  6. Unnecessary memory changes: We distinguish direct operation evaluation from downstream state-consistency evaluation. Incorrect memory modifications during ordinary interactions can affect subsequent FR and CMA results, although the current benchmark does not separately score every unnecessary storage mutation.

Below, we provide detailed responses, including the additional experimental configurations, complete results, and evaluation criteria.

Reviewer Comment R2-1

Median environment history is approximately 43.8K tokens, yet the context-based assistants use summaries and recent dialogue rather than complete histories. Without a full-history baseline, it is difficult to distinguish memory-management failures from summarization loss or contextual reasoning difficulties. The no-history versus annotated-support comparison establishes history dependence but does not resolve this distinction.

Response

We thank the reviewer for this valuable suggestion. The original base-model configuration used a rolling summary together with recent dialogue, so its performance could be affected by both summarization loss and subsequent answer generation. We have now added a full-history GPT-5-mini baseline covering all 20 environments and 4,000 evaluation instances.

For each question, this baseline receives the complete original dialogue history up to the permitted evaluation boundary, without summarization or access to future sessions. We retain the original task instructions and scoring procedures.

The complete results (accuracy, %) are:

System ConfigurationAddUpdateDeleteFRSGCDUIVISMODCMAAll
GPT-5-mini (Summary + Recent)41.8038.4048.8074.2048.2029.6046.2011.8043.0033.9542.38
GPT-5-mini (Full History Prefix)44.6039.8049.4074.8050.4039.4044.8017.4044.6038.0045.08
LILIA-Mem (GPT-5-mini)53.6040.8041.8068.2048.6043.0056.6032.4045.4045.1548.13

Providing the complete history improves overall accuracy from 42.38% to 45.08%, and CMA from 33.95% to 38.00%. In particular, Cross-device Memory (CD) improves by 9.80 percentage points, and Visibility (VIS) by 5.60 points. These results confirm that retaining the original history improves performance and that the summarization-based configuration contributed to some of the failures observed in the original experiments.

However, access to the complete history does not resolve the difficulties of memory updates and access control. Compared with the full-history baseline, LILIA-Mem improves Update/Invalidation (UI) from 44.80% to 56.60% and VIS from 17.40% to 32.40%, corresponding to gains of 11.80 and 15.00 percentage points, respectively. CMA improves by 7.15 points, and overall accuracy by 3.05 points.

We also explicitly retain and report the full-history baseline's advantages on FR, Delete, and SG rather than selectively presenting only favorable results.

These additional comparisons demonstrate that even without summarization loss, models still struggle to consistently manage evolving factual states and access permissions. Therefore, LILIA-Mem's improvements cannot be explained solely as compensation for information lost during summarization.

Reviewer Comment R2-2

The scorer–human audit contains 2,705 correct outputs among 2,800 samples, substantially different from the approximately 40–48% overall accuracies in the main experiments. The sampling procedure and representativeness need clarification. Moreover, the judge accepts 18 of 95 human-incorrect outputs. Task- and system-specific audits are needed to establish ranking reliability.

Response

We thank the reviewer for pointing out the ambiguity in our original presentation. Task accuracy and judge–human agreement measure different quantities. The 40%–48% accuracies in the main experiments measure whether the evaluated agents solve the memory tasks. Appendix B.2 instead examines whether the LLM judge assigns the same correctness label as human annotators. For example, when an agent gives a wrong answer and the judge correctly marks it wrong, that is a correct grading decision, not a correct task response. Therefore, the two rates are not contradictory.

Original sampling procedure. The 2,800 audit samples came from actual model outputs in the completed main experiments covering 4,000 benchmark questions. They were sampled with balanced coverage across household environments, evaluated models, and task categories. The original manuscript reported aggregate agreement but did not sufficiently describe this sampling process or break down grading errors by task and system.

Additional blinded audit. To address false acceptances and ranking reliability, we randomly selected 200 common questions from four household environments, with 25 questions in each of the eight subcategories. Two human annotators independently and blindly reviewed all 800 outputs from LightMem, Mem0, Dense RAG, and LILIA-Mem on these same questions. We retained the original system configurations, including the all-memory (All) settings for LightMem and Mem0. Correct, incorrect, and empty responses were all included.

Task-specific audit results (rates in %):

TaskOutputsFalse-positive rateJudge–human agreement
MOD3002.5096.33
FR1000.00100.00
CMA-SG1003.4596.00
CMA-CD1003.3996.00
CMA-UI1004.7695.00
CMA-VIS1004.3595.00

Of the 800 outputs, 771 grading decisions agreed with human judgments (96.38%), with 14 false acceptances and 15 false rejections. Across LLM-judged categories, false-positive rates range from 2.50% to 4.76%. FR uses option matching and achieved full agreement with human judgments in this audit.

System-specific audit and ranking (same 200 questions per system):

SystemOutputsFalse acceptancesJudge rankingHuman ranking
LightMem (All)200444
Mem0 (All)200433
Dense RAG (Top-10)200322
LILIA-Mem200311

The ranking is unchanged after human review: LILIA-Mem > Dense RAG > Mem0 > LightMem. LILIA-Mem also achieves the highest human-assessed accuracy on this shared question set. Thus, the observed grading disagreements do not alter the relative ordering of these four systems in the additional audit.

Reviewer_d6i2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates long-term memory in multi-user, multi-device assistant settings. It first proposes a benchmark that has shared memory across different users and devices and tests how the system revises memory, recognizes current facts, and applies evidence to requests. Then, it proposes a memory management framework tailored to this setting and demonstrates improvements over existing baselines on the proposed benchmark.

Critical strengths
  1. The paper open-sourced a multi-user, multi-device benchmark, which might be useful for future research within the community.

  2. The benchmark evaluates three different tasks and tests different memory abilities: memory operation decision, which checks whether the system can revise memory correctly; factual memory recognition, which checks whether the system can recognize current facts; and contextual memory application, which checks whether the system can apply evidence to requests.

Critical weaknesses

I think there are multiple weaknesses:

  1. First, the presentation of this paper needs substantial improvement, especially the figures. The figures in the paper contain too much small-font text and too much information without highlighting the key points. It is very hard to understand these figures at first glance. Also, there are multiple places in the figures where the text overflows beyond the boundaries.

  2. My major concern with this paper is its motivation. Based on my understanding, the user and device identities are provided for each session, so I'm confused about why shared memory is necessary in this scenario rather than maintaining separate memory for each user.

  3. The interactions across users are very simple, consisting only some fact-checking of other users' information. Compared to previous related work, I couldn't see the novelty of this paper.

  4. The benchmark is fully synthesized by LLMs, with dialogue sessions generated by an LLM from predefined structured events. I think the benchmark doesn't capture the complexity of real-world multi-user and multi-device use cases.

  5. The proposed system only applies some very specific and well-defined rules tailored to the benchmark design. Even with these rules, the improvements are marginal. I don't see any novelty in the proposed system.

Questions for authors

Please see the Weaknesses section, especially point 2) ~ 5).

11 comments

Overall Response to Reviewer R1

We thank the reviewer for the feedback. However, we believe that several major concerns raised in the review reflect an insufficient understanding of the practical requirements and existing industrial practices of real-world multi-user, multi-device AI assistants. In particular, the review appears to assume that knowing user and device identities is sufficient to manage persistent memory, that everyday short-form interactions imply simple memory requirements, and that deterministic state constraints necessarily constitute benchmark-specific rules. These assumptions overlook the fundamental challenges encountered when a persistent AI assistant serves multiple users across personal and shared devices.

In real-world deployments, users continuously interact with the same assistant through smartphones, smart speakers, in-car systems, and other devices, while multiple household members may also share individual devices. Under these conditions, memory management is not simply a matter of storing and retrieving facts by user ID. The system must determine who a fact concerns, whether it remains valid after subsequent revisions, under which conditions it applies, and whether the current requester is authorized to access it. These are practical requirements rather than artificial complications introduced by our benchmark. Our anonymized real-world assistant logs and documented industrial reliability challenges provide direct evidence of their relevance.

We therefore clarify five points:

  1. Presentation: We acknowledge the figure readability issues and have improved the layouts, typography, visual hierarchy, and text boundaries.
  2. Motivation: Per-user storage partitioning does not eliminate the need for authorized cross-user information sharing, cross-device state continuity, or revision-aware permission preservation.
  3. Benchmark novelty: LILIA goes beyond simple cross-user fact-checking by jointly evaluating memory operations, valid-fact recognition, and contextual application over continuously evolving multi-user, multi-device histories.
  4. Benchmark realism: LILIA is grounded in real-world assistant interaction patterns. Structured events and constrained dialogue generation provide traceable reference states and reproducible evaluation without assuming that real-world interactions are inherently long or structurally explicit.
  5. Methodological contribution: LILIA-Mem is not a hard-coded rule system. It combines LLM-based semantic relation inference with constrained state revision and semantic applicability verification. Controlled ablations demonstrate substantial performance degradation when these mechanisms are removed, particularly on memory updates and visibility-sensitive tasks.

We address each concern below with concrete evidence, experimental results, and comparisons with existing research and industrial practice.

Reviewer Comment R1-1

First, the presentation of this paper needs substantial improvement, especially the figures. The figures in the paper contain too much small-font text and too much information without highlighting the key points. It is very hard to understand these figures at first glance. Also, there are multiple places in the figures where the text overflows beyond the boundaries.

Response

We thank the reviewer for highlighting the readability issues in our figures. We acknowledge that several figures in the original manuscript contained excessively small text, dense information, insufficient visual emphasis, and text overflowing beyond figure boundaries.

We have revised the relevant figures by increasing font sizes, simplifying textual annotations, and reorganizing the layouts to establish a clearer visual hierarchy. We have also emphasized the core workflows and key components while correcting the text overflow issues.

Reviewer Comment R1-2

My major concern with this paper is its motivation. Based on my understanding, the user and device identities are provided for each session, so I'm confused about why shared memory is necessary in this scenario rather than maintaining separate memory for each user.

Response

We thank the reviewer for raising this important question. The concern appears to conflate physical storage partitioning with the logical requirements of cross-user memory governance. LILIA does not require a single physically shared database. Rather, "shared memory" refers to the coordinated management of private and shareable information by a persistent assistant serving multiple users across multiple devices.

User and device IDs identify the current interaction context, but they do not determine which historical facts may be accessed, whether those facts remain valid, or how new statements should revise existing memories.

First, the requester and the subject of a fact are not necessarily the same person. Strictly isolating memory by requester ID would prevent legitimate cross-user information sharing. LILIA's Visibility (VIS) task includes 240 instances requiring authorized disclosure and 260 requiring nondisclosure, explicitly evaluating both information sharing and privacy protection.

If separate per-user memories were accessible only to their respective owners, the system could not correctly answer the 240 legitimate cross-user queries. Conversely, once controlled cross-user access is permitted, the assistant must determine fact ownership and access authorization, regardless of whether the underlying storage is physically partitioned. Thus, per-user partitioning is a valid implementation choice, but it does not eliminate the need for memory governance.

Second, cross-device revisions must preserve access restrictions that have not been explicitly changed. Consider the example in our Introduction: Alice states on her phone that she likes jazz and requests that this preference remain private. She later changes her preference to classical music using her laptop, without modifying the privacy restriction. The assistant must update the preference, preserve its private visibility, and refuse to disclose it when Bob asks through a shared speaker.

Alice's user ID helps locate the relevant memory, but it cannot determine which fields the new statement modifies or which existing restrictions must remain unchanged. These decisions require semantic interpretation of the update and its relationship to the existing memory. Consequently, revision-aware preservation of access conditions remains necessary even with separate per-user storage.

Third, device identity indicates the interaction source, not automatically the applicability or accessibility of a memory. Personal information without device-specific restrictions should remain available when its owner switches devices. Conversely, using a shared device must not grant other users access to private information.

LILIA separates these requirements through controlled tasks. Cross-device Memory (CD) holds the user and target fact fixed while changing the query device to evaluate memory continuity. Visibility (VIS) holds the source and query devices fixed while changing the requester to evaluate authorized disclosure and privacy protection. These tasks specify behavioral requirements without prescribing a particular storage architecture.

Real-world evidence further motivates these distinctions. Google's technical report, Building Personal Intelligence: A Step Towards Truly Personal AI, identifies attributing another person's preferences to the current user, missing subsequent cancellations, and ignoring user corrections as known reliability challenges [1].

These observations illustrate why access to historical records alone does not guarantee correct attribution, validity tracking, or contextual use. LILIA addresses precisely these challenges, independently of whether the underlying database is shared or partitioned.

Reference

[1] Google. Building Personal Intelligence: A Step Towards Truly Personal AI. January 2026, pp. 10–11.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

We have provided point-by-point responses, additional experiments, and manuscript revisions addressing the reviewers’ concerns. We summarize the main contributions and new evidence below.

1. Practical motivation and benchmark contribution.

Known user and device identities do not resolve authorized sharing, evolving facts, or preservation of access restrictions during updates. Brief requests are realistic: Bentley et al. [1] report a four-word median command length across 65,499 smart-speaker interactions from 88 households. Google’s Personal Intelligence report [2] documents preference misattribution, missed cancellations, and overlooked corrections.

Informed by anonymized assistant interaction patterns, LILIA constructs event-grounded histories in which ordinary interactions are interleaved with persistent memory changes. Its 20 environments, 16,000 sessions, and 4,000 questions connect memory-operation decisions (MOD), factual recognition (FR), and contextual application (CMA) to traceable reference states. This enables analysis of the gap between recognizing facts and applying them under evolving constraints. The eight question categories are deliberately balanced, with 500 instances each, for diagnostic coverage; their frequencies are not estimates of production traffic.

2. Additional controlled experiments.

  • Full history (4,000 instances; R2-1): GPT-5-mini achieves 45.08% overall accuracy and 38.00% CMA with complete permitted histories, compared with 48.13% and 45.15% for LILIA-Mem. Summarization loss alone therefore cannot explain the performance difference.
  • Matched final evidence budgets (800-instance subset; R2-3): With at most 10 records and 4,096 tokens and native baseline retrieval, LILIA-Mem achieves 47.88% overall accuracy, versus 45.00% for Dense RAG and 45.13% for Mem0. Its overall and CMA advantages persist across three independent runs on this subset.
  • Multi-device extension (300 additional instances; R2-4): The extension covers continuity, combined constraints, and device-dependent answers. Removing structured device metadata reduces accuracy on device-dependent questions from 46% to 28%.

3. Methodological evidence and evaluation reliability.

LILIA-Mem combines semantic relation inference, constrained revision, and applicability verification. Controlled ablations support these components . On the full benchmark, its gains over Dense RAG are 2.17 percentage points overall, 3.97 on CMA, and 12.32 on Update/Invalidation. CMA gains over the original Mem0 and LightMem configurations are 9.50 and 9.96 points, respectively.

An additional blinded audit of 800 outputs achieves 96.38% judge–human agreement and preserves the ordering of four systems on 200 common questions . We also clarify MOD’s scoring scope and report same-model computational costs, including the additional overhead relative to RAG.

Figures 1 and 2 have been revised in the uploaded PDF. We respectfully ask that the benchmark design, controlled comparisons, and component evidence be considered together in the final assessment.

Thank you for your consideration.

References

[1] Bentley, F., LuVogt, C., Silverman, M., Wirasinghe, R., White, B., and Lottridge, D. Understanding the Long-Term Use of Smart Speaker Assistants. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3), Article 91, 2018, pp. 1–24. https://doi.org/10.1145/3264901

[2] Venkatachary, S. Building Personal Intelligence: A Step Towards Truly Personal AI. Google, January 2026, pp. 10–11. https://ai.google/static/documents/building_personal_intelligence.pdf

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.