acceptodds
Desk rejected at ICLR 2027

One Assistant, Many Memories: Benchmarking Real-World Group Assistance When Recall Is Not Enough

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The value of memory lies not only in recall but in using it to provide appropriate assistance. Recent benchmarks test memory use in personalisation and agentic tasks, but seldom evaluate assistance without fixed answers or integration across multiple participants. Group chats make both challenges more acute: an assistant must integrate information from multiple members, track revisions and conflicts over time, and produce one response for the group. Yet existing group memory benchmarks rely on synthetic conversations and inherit constrained evaluation from general memory benchmarks, focusing on recall against fixed answers or decisions over a closed set. We introduce GroupAssistBench, the first memory benchmark from real-world group conversations with a deployed group assistant, covering 80,299 messages in 50 groups and costing over $53K. Over the same histories, GroupAssistBench evaluates memory recall (L1) with reference answers and memory use (L2) with service requests without fixed answers across single, distributed, and conflicting memories (M1–M3). Requests are verified to require group history, memories are anchored to their sources, and L2 uses tailored rubrics with penalties derived from real user feedback. Across evaluations of frontier models with full context, retrieval methods, and memory systems, the strongest memory system rivals full context on recall but falls far behind on use, especially for distributed and conflicting memories. Together, two additions that relate retrieved memories and bind remembered facts as response constraints bring consistent gains, showing that group memory systems must improve use, not recall alone.

Primary area:
datasets and benchmarks
Keywords:
Group Chat, Agent Memory, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_15eAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GroupAssistBench, a benchmark constructed from real-world group chats. The authors organize the benchmark along two dimensions, memory recall (L1) and memory use (L2), and three types of memory: single, distributed, and conflicting memories (M1–M3). Beyond traditional fixed-answer tasks that evaluate the retrieval capabilities of memory systems, the benchmark also emphasizes the practical value of memory use. Specifically, it includes tasks that assess how systems leverage conflicting and updated information in multi-party conversations to answer open-ended questions, such as planning a trip. The paper is well motivated and addresses a problem with clear practical relevance. The authors conduct extensive experiments on existing approaches, including full-context, RAG-based, and external memory systems. The results reveal that current memory systems consistently struggle with memory use, providing valuable insights and directions for future research. Furthermore, based on the limitations identified through their experiments, the authors propose two intervention strategies, RER and ACB, both of which demonstrate effective improvements over the existing baselines.

Critical strengths
  1. The paper is well motivated and has strong practical relevance. The authors focus on group conversations and specifically design tasks to evaluate memory use in open-ended scenarios, which is particularly valuable for real-world agent applications. Moreover, the evaluation dataset is constructed from group chat records collected in real-world deployment settings, making the benchmark more representative of practical human-AI interaction scenarios.

  2. The authors evaluate several existing approaches, including full-context, RAG-based, and external memory systems. Their results reveal that even when existing memory systems perform well on memory retrieval, they still exhibit significant limitations in memory use. The authors further analyze the underlying reasons for this gap and propose two targeted intervention methods, RER and ACB, both of which demonstrate tangible improvements over the baselines.

Critical weaknesses
  1. The authors conduct a human evaluation in Appendix C, but they acknowledge that completeness is not effectively assessed. For group-oriented tasks, incorporating the perspectives and preferences of all group members is also an important factor, which may affect the evaluation of memory systems' effectiveness.

  2. The paper evaluates the ability to handle conflicting information through M3. However, according to Appendix A, its construction primarily focuses on cases where information has been historically updated and the latest state has already been confirmed. In real-world scenarios, group conversations may also involve differing preferences and requirements among multiple members, raising the open-ended challenge of how an agent can leverage memory to create plans that accommodate the needs of different group members. I believe incorporating such tasks would provide a more comprehensive assessment of the challenges associated with memory use in real-world group assistance.

Questions for authors
  1. Regarding Weakness 1, the authors could conduct additional experiments to further validate the evaluation of completeness in the benchmark.

  2. Regarding Weakness 2, the authors could consider incorporating the aforementioned scenario into the benchmark.

5 comments

We thank Reviewer 15eA for the thoughtful feedback. Revisions addressing this review are highlighted in orange in the revised manuscript. We address each concern below.

[W1 & Q1] Completeness of the required memories, and whether the perspectives of all members are covered.

We would like to clarify that our construction pipeline first extracts candidate facts needed for the response from the history before the cutoff, then removes candidates whose absence would not change the response (Appendix A.3). Retained anchors are further checked for necessity during quality control (Appendix A.6). To complement these checks and the human validation in Appendix C, we conducted a supplementary audit of the coverage of relevant facts and member perspectives using the same three annotators and 60 L2 instances.

Method. Reading the whole history for every instance is impractical, so we used the 14 evaluated systems as the search: every message that any system's response explicitly cited and that is not an anchor of the instance became a candidate missed fact (668 candidates over 54 of the 60 instances; up to 10 per instance were adjudicated, giving priority to messages cited by several systems, 355 in all). For each candidate the annotators saw the request, the anchors, the positive criteria, and the candidate message with one message of context on either side, and judged whether it is a preference, constraint, fact, or decision that a good response should use and that the existing anchors and criteria do not cover. Pairwise agreement is 0.83–0.88. This search gives a lower bound on misses in general, but it is the right bound for the question at hand: a missed fact that no system uses cannot change any system's score, so the facts systems actually cited are exactly the misses that could affect the comparison.

Results. By majority vote, 19 of the 355 candidates (5.4%) are relevant facts that the rubric does not cover. They fall in 12 of the 60 instances (20%; 3 instances if unanimity is required) and concentrate in M3 (8 of 20 instances, against 1 of 20 for M1 and 3 of 20 for M2): M3 requests ask how a decision was reached step by step, so intermediate steps beyond the anchored decision points are arguably relevant. On member perspectives, 8 of the 19 are earlier replies of the assistant that the group adopted, and all 11 missed member statements come from members who are already represented among the anchors of that instance; no instance omits a member whose statements matter. Together with the construction statistic that 56% of M2 and 93% of M3 instances draw anchors from at least two members, the perspectives of the members involved are covered.

M1M2M3All
Instances20202060
Candidates adjudicated60144151355
Confirmed missed facts (majority vote)141419
Instances with ≥1 missed fact (majority / unanimous)1 / 03 / 28 / 112 / 3

Effect on the evaluation. A fact missing from the rubric cannot be rewarded, so a response that uses it receives no credit and the automatic scores are lower bounds; since the candidates come mostly from the explicit citations of the full-context models, the under-crediting falls on the leading systems rather than favouring them. Adding the 19 facts to the rubrics as grounding criteria and crediting the systems that cited them changes the mean score of every system on the 60 instances by less than 0.02 and changes no rank, except a swap of the two leading full-context models that the clustered bootstrap already marks as inseparable (credit by citation is a proxy that favours the full-context models, which cite explicitly); even crediting every system for every missed fact could raise a mean score by at most 0.05, below the gaps between paradigms.

[W2 & Q2] Accommodating different members' preferences and requirements. Thank you for highlighting this aspect. L2 M2 and M3 accommodate different members' needs through planning, resource allocation, recommendations, and communication, not only historical updates. Figure 1 combines the revised April 30 departure, extra ¥50 per person, a window-seat preference, and costs for nine travellers. The L2 M3 case in Appendix G combines a three-minute duration limit with another member's concern about video completion rate. The existing L2 M2 instance below illustrates simultaneously valid preferences.

L2 M2: A travel budget accommodating different priorities

Request: “Suggest a compromise for our travel budget that meets both sides' needs as far as possible.”

Memory anchors (A–F denote members):

  1. A1. A proposes a basic budget of ¥2,200 per person for meals and accommodation.
  2. A2. B prefers ¥2,500 for unusual activities and spontaneous additions.
  3. A3. C also prefers ¥2,500 and does not want both photography and dining reduced.
  4. A4. D requires shared and personal expenses to be distinguished.
  5. A5. A rejects sharing personal shopping costs by default.
  6. A6. E warns that averaging to ¥2,350 may satisfy neither side.
  7. A7. B considers the contingency reserve inadequate for nine travellers.
  8. A8. F suggests a basic package with optional upgrades rather than one budget for everyone.

Rubric (translated and abridged):

  • G1, +3. Recall concerns about a numerical compromise and the inadequate reserve.
  • S1, +1. Propose a shared basic budget for essentials and voluntary upgrades, explaining how both sides' needs are accommodated.
  • G2, +1. Recall F's basic package and A's rule against sharing personal shopping costs by default.
  • S2, +2. Specify a usable reserve limit, spending conditions, and approval rules for extra activities.
  • G3, +2. Recall C's photography and dining requirements and D's distinction between shared and personal expenses.
  • S3, +2. Separate shared costs, optional upgrades, and personal purchases in the payment plan.
  • P1, −2. Penalize misremembering the original ¥2,200 basic budget.
  • P2, −2. Penalize changes that violate the advance communication agreement.

The response must turn different members' priorities into an executable budget and payment arrangement, not merely summarize their opinions.

Evaluation on requests involving multiple members. Reviewing all evaluated L2 M2 and M3 item cards identified 175 requests across 36 chats (124 M2, 51 M3), or 17.2% of evaluated L2. The tables report request types, subset scores, and rankings.

Request typeHow members' needs or positions are addressedRequestsM2M3
Shared planning and allocationDevelop a common plan or allocate resources to accommodate different members' needs78717
Joint communicationAddress multiple members' different states, concerns, or requirements in one response49454
Applying or arbitrating group disagreementsApply agreed outcomes or reconcile differing accounts from multiple members48840
Total17512451
MethodAll L2 Avg.Subset M2 (124)Subset M3 (51)Subset Avg. (175)Subset rank
Full Context
Claude Sonnet 50.4280.3590.4580.3882
DeepSeek-v4-flash0.4690.4090.5230.4421
Qwen-3.5-9b0.3480.2860.2310.2706
GPT-5.6-terra0.4310.3620.2690.3353
Retrieval Augmented
BM250.3700.3270.1960.2894
Dense retrieval0.3620.2780.2600.2735
GraphRAG0.3440.2580.2260.2498
RAPTOR0.3340.2750.2160.2587
External Memory
Mem00.2450.1670.1840.17212
LightMem0.1390.0990.1040.10014
A-MEM0.2350.1540.1770.16113
MemoryOS0.2690.2120.1740.20110
xMemory0.2030.1770.2470.19711
EverMemOS0.2490.2080.2820.2309

All 14 systems score lower on this subset than on all L2. Rankings remain close to Table 3 (Kendall correlation 0.89, maximum shift two places). On M3, EverMemOS and xMemory score 0.282 and 0.247. EverMemOS's point estimate exceeds GPT-5.6-terra Full Context (0.269). This suggests potential for hierarchical organization to integrate members' evidence when resolving conflicts. The requests assess applying memory to differing member needs. We have clarified this coverage under the Memory axis in Section 3.1 (Task Formulation). Appendix F.6 (Assistance Involving Multiple Members) adds the planning example, request distribution, and subset results. These additions are highlighted in orange.

Dear Reviewer 15eA, thank you again for your valuable feedback. As the discussion period is nearing its end, we would appreciate hearing whether our responses have addressed your main concerns. If any part remains unclear or if you have any further questions, we would be happy to clarify anything further.

Reviewer_H2LhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GroupAssistBench to evaluate whether assistants can use group conversation history for open-ended assistance, beyond answering recall questions. It has 2,395 questions grounded in 50 real group conversations. These cover single memories, combinations of memories, and resolved updates or conflicts. The authors compare 14 configurations and track 497 memories across recall and assistance tasks. They also test interventions for organizing evidence and adding constraints during generation. Validation includes human checks of the instances, rubrics, and automated judgments.

Critical strengths

The main strength of the paper is that the dataset comes from conversations with a deployed group assistant and includes native assistance requests. This makes it possible to study how assistants handle information from different members, their agreements, and changing circumstances.

The comparison across 14 configurations and the paired-memory analysis are also useful. I find the changes in system rankings between recall and assistance worth looking into.

I also like the validation of the automated evaluation through human checks.

Critical weaknesses

My main concern is whether L2 consistently measures appropriate memory use. Table 18 requires the response to explicitly restate the relevant fact. A response can behave appropriately and still get no grounding credit if it does not restate that fact. This seems to conflate not mentioning a memory with not using it. The judge's grounding recall is also only 0.67. The authors acknowledge that their unused-memory estimates are upper bounds, and human scoring keeps the same ranking for the three sampled systems. But this still leaves the question of what should count as successful memory use.

I also think the paper needs to be clearer about what is new relative to the closest benchmarks. StreamMemBench already studies failures to use stored evidence, while ContextWeave evaluates memory through downstream task performance. Neither is discussed. I think the distinctive contribution is extending evaluation to real group assistance. The broader argument about recall versus use already has substantial precedent.

Questions for authors

Can you separate responses that use a memory appropriately without restating it from responses that actually fail to use it? How would this affect L2 scores and the present but unused analysis?

What is different in your evaluation and diagnostic design compared with StreamMemBench and ContextWeave, beyond the group-conversation setting?

What evidence distinguishes RER’s recovery of missing information from better organization of information already available, especially given Table 15 and the unavailable turn-level coverage for LightMem?

5 comments

We thank Reviewer H2Lh for the valuable feedback. Revisions addressing this review are highlighted in purple in the revised manuscript. We address each concern below.

[W1] What should count as successful memory use, and whether the strict grounding criterion and the judge's 0.67 grounding recall bias L2.

We first clarify successful memory use (1), then address the judge's grounding recall (2).

1. Definition. We would like to clarify that in a group, a memory is successfully used only when the members can see which member's statement or constraint the response rests on. Group chat is public: a reply is read by everyone, members other than the requester must be able to check that their constraints were honoured, and a recommendation whose basis is invisible cannot be corrected by the person it concerns. Stating the grounding basis is therefore part of the service, not a proxy for it, which is why grounding criteria require the fact to appear (Table 18). The alternative admits coincidence: a reply that merely agrees with a constraint cannot be told from a safe default. Three actual responses illustrate this distinction. The request asks the assistant to schedule tonight's stream ("keep it short; the streamer said he does not want it to run past 10:30"); the grounding criterion requires two earlier statements, the streamer's "can we end at ten? don't run over again" (#723) and the manager's "control it yourself, don't run over once you get chatting" (#724).

ResponseSystemExcerpt (translated, abridged)Judge
A. States the basis and acts on itDeepSeek-v4-flash (full context)"Scheduled per the streamer's request, not too packed and nothing past 10:30 … 21:55–22:00 wrap-up, off air at 22:00 sharp … total length capped at two hours; the streamer keeps control of the room and does not run over once the chat warms up, as the manager reminded before."met
B. Acts consistently, basis not shownBM25"21:52–22:00: a fixed sign-off buffer, the last song or a goodbye, no new topics … the key is no new song requests and no new topics after 21:40, then close at 22:00, so it will not run to 10:30."not met (audit class B)
C. Conflicts with the constraintGraphRAG"22:00: say goodbye on time … even if the chat is lively, extend at most to 22:10, not to 22:30."not met (audit class C)

Both A and B end the stream at 22:00. The difference is not the end time but whether the grounding basis is visible. A ties the plan to the streamer's request and the manager's earlier reminder. B gives a compatible schedule with concrete safeguards but names neither statement, so the group cannot tell it from a generic routine. C allows an extension to 22:10, within the 22:30 limit the request mentions but against the earlier 22:00 constraint. Grounding separates A from B, and constraint compliance separates B from C. (The example illustrates the response types, not retrieval.) We also examined this pattern across the benchmark, comparing responses for 384 paired memories present in some systems' inputs but absent from others on the same instance. Replies that state the memory depend on its availability: 54.4% when it is present, 9.6% when it is absent. Replies that are merely consistent with it without stating it do not: 10.6% when present, 14.7% when absent. A reply that agrees with a constraint without naming it is therefore as likely to have been produced without the memory as with it, which is why consistency cannot count as use. Acting on a fact without stating it is rare: among the 50 grounding criteria that the judge missed but the annotators accepted, only 8, 2.3% of the 351 grounding cells, do so. The revision adds this and a second real case to Appendix C.

2. The judge. Human validation shows that the judge is conservative on grounding, and in one direction only. Against the annotators' majority vote, its grounding precision is 1.00 and recall is 0.67: it never credits a grounding criterion that the majority rejects, and it withholds credit for some that the majority accepts, usually when the response paraphrases the fact rather than stating it (42 of the 50 withheld criteria are explicit statements or paraphrases of the fact, Appendix C.4). Across all criterion types, judge–human agreement is 89.3%, the same level as the agreement between annotators (89.8%). The judge's mean rubric score is 0.13 below the human majority, but the ranking of the three evaluated systems is identical under the judge and under every annotator. Because the same rule and the same judge apply to every system, this offset lowers all scores alike and cannot favour one system over another: it makes the L2 scores conservative, not biased (Appendix C.3).

We have clarified the rationale for requiring explicit grounding in Section 3.3 (Evaluation). This clarification is highlighted in purple in the revised manuscript.

[W2 & Q2] What is new relative to StreamMemBench and ContextWeave, and how do the evaluation and diagnostic design differ beyond the group setting. We thank the reviewer for highlighting these related works and encouraging a clearer comparison. We would like to clarify that ICLR does not require comparisons with work available only on arXiv or published at a peer-reviewed venue on or after July 17, 2026. ContextWeave appeared on arXiv in August 2026, while StreamMemBench was published in Findings of EMNLP in October 2026. Although these comparisons are not required, we provide them below to address the reviewer's question and clarify the distinct contributions of GroupAssistBench. The gap between recall and use is also studied in related work. But GroupAssistBench focuses on measuring this gap in real group assistance, using conversations with a deployed assistant and evaluation criteria derived from real member feedback.

Beyond the setting, the table below compares four aspects of the evaluation and diagnostic design:

AspectGroupAssistBenchStreamMemBenchContextWeave
Source of criteriaGrounding criteria are anchored to verbatim group messages, and penalty criteria are derived from 97 real member corrections (Appendix B).Tasks and feedback are generated by an LLM over a lifelog. Responses are judged pass or fail against the anchor by an LLM user simulator.Tasks are reconstructed from real work histories. Task-specific rubrics are written for the benchmark.
Memory dependencyEvery instance is typed by what it requires: one memory (M1), memories distributed across members (M2), or conflicting memories (M3), and all results are reported per type.One anchor per task, followed through initial use, feedback, and reuse. No typing by number of memories or by conflict.Tasks are grouped by activity type and by distance to the relevant earlier task. No typing by memory dependency.
Recall paired with use497 individual memcall probe and an assistance request over the same history, and retrieval logs split unused memories into not retrieved and present but unused.Preservation of each anchor is checked in the written memory before the task, then its use in the response. The two rates are reported separately.The same task is run with and without recall, with diagnostics of the task level.
Judge validationThree annotators, 1,152 criterion judgments each, giving errors per criterion type and their direction, score agreement, and an identical ranking of three systems.Human agreement is reported on sampled trajectories for the four metrics.Agreement with sampled human task level. No criterion-level validation is reported.

Three aspects are central to our findings. Using real group conversations from a deployed assistant grounds the evaluation in actual interactions among members and the assistant. Reporting results separately for M1, M2, and M3 makes performance changes across memory dependencies explicit. Deriving penalty criteria from real member feedback connects evaluation to specific failures that members identified in deployed assistance.

We have clarified the contribution in Section 1 (Introduction), added a discussion of StreamMemBench and ContextWeave in Section 2 (Related Work), and included both benchmarks in Table 1. These revisions are highlighted in purple in the revised manuscript.

[Q1] Separating responses that use a memory appropriately without restating it from responses that fail to use it, and the effect on L2 scores and on the present-but-unused analysis. The rubric already separates them, because it scores the two things with different criteria. Grounding criteria score whether the response states the fact it relies on; required service criteria score whether it does what the request needs, and penalty criteria score failures of memory use (Table 18's "must state the fact" rule applies to grounding criteria only). A response that applies a memory without restating it therefore keeps its service credit and incurs no penalty; it loses only the grounding points. A response that actually fails to use the memory also fails the service criteria that depend on it or triggers a penalty. We cross-classified the 14 × 1,019 L2 responses by whether every grounding criterion is met (the fact is stated) and whether the behaviour is appropriate (every required service criterion met and no penalty triggered):

ResponsesShareMean score
States the fact, behaviour appropriate9%0.98
States the fact, behaviour not appropriate10%0.60
Not state the fact, behaviour appropriate15%0.49
Not state the fact, behaviour not appropriate66%0.14

The first category is 15% of responses (8–22% per system, with no paradigm systematically higher) and already receives about half of the score; the second is 66% and receives 0.14. Because the service and penalty scores are untouched, the first category can lose only the grounding points, so its share bounds the effect on both the L2 scores and the present-but-unused counts. What the first category cannot show is whether the memory was used at all. Consistent-but-unstated replies occur in 10.6% of the cases where the memory was in the system's input and in 14.7% where it was not, so from the response alone a correct application cannot be told from a lucky guess; by human coding, only 2.3% of grounding judgments are genuine use without statement. This is the point of W1: in a group, a reply that shows its basis but misses a requirement can be checked and corrected by the member it concerns, whereas an appropriate reply with no visible basis cannot be, and would have scored the same had it been a guess. The rubric therefore ranks the former above the latter by weight, not by exclusion, since the third cell keeps its service credit. The revision reports this cross-classification in Appendix C.4.

Reviewer_DvV7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GroupAssistBench is proposed to evaluate the group memory where accurate retrieval and solving conflictions are challenging. The bench is constructed from real-world group conversations with a deployed group assistant, covering 80,299 messages in 50 groups.

Critical strengths
  1. Reframes memory evaluation from "storing the right fact" to "acting on it correctly for the right person", which is the failure mode that actually matters in multi-user deployment.
  2. Grounded in real conversation records rather than synthetic templates, preserving messy, scattered, and contradictory evidence.
  3. The L1/L2 split cleanly separates retrieval quality from constraint application, exposing that strong recall does not imply useful assistance.
Critical weaknesses
  1. No counterfactual or leave-one-out control isolating whether L2 failures stem from missing evidence, retrieval ordering, or the model's instruction-following.
  2. Evaluated systems are largely first-generation memory architectures not designed for multi-principal access; results may understate what purpose-built group memory could achieve.
Questions for authors
  1. The 2,395 instances are spread over six cells (L1/L2 × M1/M2/M3), and Table 3's rankings are read cell by cell. Please give the exact count per cell and the number of distinct source chats contributing to each. Without this, a rank gap of one or two positions cannot be distinguished from sampling noise.
  2. Table 3 shows rank superscripts and pairwise-gain markers but no confidence intervals, while Tables 4–5 do report them (±). Please add bootstrap CIs clustered at the chat level; instances drawn from the same 50 chats are not independent, and level differences of ~0.02–0.04 are being used to order systems.
  3. M1 is "single-user memory", but M2/M3 require evidence from several members. Did each member receive a separate store, one shared store, or a store with speaker metadata? This choice largely determines whether the baseline comparison is fair, and it is not visible in the main text.
5 comments

We thank Reviewer DvV7 for the constructive feedback. Revisions addressing this review are highlighted in blue in the revised manuscript. We address each concern below.

[W1] Sources of L2 failures. We selected one representative from each of the three categories: Full Context, Dense retrieval for RAG, and xMemory for external memory. We ran three controls with the same answering model (GPT-5.6-terra) and judge (GPT-5.6-sol), using the same questions across conditions within each system:

  • Oracle: Supplement missing supporting evidence for Dense retrieval and xMemory. For Full Context, which already has access to the chat history, remove unrelated history.
  • Random reordering: Randomly shuffle the evidence blocks using three fixed seeds and average the three scores, without changing the evidence itself.
  • Instruction following: Keep the evidence unchanged and append an explicit instruction to apply the relevant members' preferences, constraints, and prior decisions. The instruction is:

Before answering, identify the preferences, constraints, and prior decisions in the provided information that concern the requester and other members involved in this question. Your answer must explicitly apply these specific facts (names, numerical values, dates, and key points of the original statements), rather than offer generic advice unrelated to member information. When relevant information is available, do not decline to answer on the grounds of “insufficient information” or “no relevant information.”

SystemLevelBaselineOracleRandom reordering (mean of 3 runs)Instruction following
Full ContextM10.5040.4880.4860.549
M20.4350.4740.4320.505
M30.3050.4430.3270.372
Avg.0.4310.472 (+0.042)0.428 (−0.003)0.495 (+0.064)
Dense retrievalM10.4280.4690.4140.491
M20.3680.4310.3570.401
M30.2420.3550.2640.271
Avg.0.3620.428 (+0.066)0.355 (−0.007)0.401 (+0.039)
xMemoryM10.1340.1680.1330.212
M20.2260.3080.2280.269
M30.2220.3180.2210.233
Avg.0.2060.279 (+0.074)0.206 (+0.000)0.252 (+0.046)

The results are summarised above. Oracle evidence and explicit instructions improve average L2 scores across all three systems, whereas random block reordering changes them little. These results point to two sources of failure: missing or obscured supporting evidence, and failure to apply available evidence. Presentation order appears less influential under the tested perturbation.

For Full Context, M1 requires a single memory that a capable model can often identify without narrowing the history. Removing unrelated evidence may therefore offer limited benefit. Its Oracle score remains close to Baseline (0.488 vs. 0.504). M2 requires integrating multiple memories, and M3 additionally requires resolving updates, making focused evidence more helpful. Oracle helps the model find relevant constraints, while the added instruction directs it to apply those constraints. L2 requests do not restate all the preferences and prior decisions that should shape the response. The instruction makes using these facts an explicit requirement when deciding what to recommend and for whom. This can benefit all three systems because a generally plausible answer may still overlook the needs of particular members.

Human validation of oracle evidence and benchmark answerability. In addition, to verify that the oracle evidence supports the expected responses, the three annotators from Appendix C assessed whether each of the 277 positive rubric criteria across its 60 L2 instances can be satisfied from the Oracle window alone (the establishing message of each required memory with two messages on either side, plus the request). Pairwise agreement is 0.95–0.99.

M1M2M3All
Positive criteria76101100277
Satisfiable from the window (majority / unanimous)74 / 66100 / 9799 / 99273 / 262
Attainable rubric score S̄_max (majority / unanimous)0.99 / 0.900.99 / 0.970.99 / 0.990.99 / 0.95
Instances attainable in full18 / 2019 / 2019 / 2056 / 60
Oracle score of Full Context (table above)0.490.470.440.47

98.6% of the positive criteria are satisfiable from the window by majority judgment; the remaining 1.4% are three service criteria and one grounding criterion that need information outside it, and none is unsatisfiable or ill-posed, which means most instances are answerable, and the models fail to fully utilise the evidence. We have added a summary of the diagnostic experiments and human answerability assessment to Section 4.2, with detailed experiments in Appendix F.4 and human validation in Appendix C.4. These additions are highlighted in blue in the revised manuscript.

[W2] Memory systems designed for group conversations. We sincerely appreciate this suggestion. Before presenting the additional comparison, we would like to clarify that, to our knowledge, we had not identified a publicly available framework specifically optimized for group memory before the July 17, 2026 cutoff in ICLR's comparison policy. To address this concern, we added SpeakerMem-R1, a framework specifically designed for memory in group conversations that first appeared on arXiv on September 22, 2026.

SpeakerMem-R1 uses two retrieval tracks combining original messages indexed by speaker with derived memories about individuals and groups. We evaluate its LLM Writer configuration with GPT-5.6-terra for writing, retrieval, and answering under the same evaluation protocol. The table extends Table 3 with SpeakerMem-R1 and updates the ranks across all 15 methods.

MethodMemory typeL1 M1 metL1 M1 F1L1 M2 metL1 M2 F1L1 M3 metL1 M3 F1L1 Avg. metL1 rankL2 M1L2 M2L2 M3L2 Avg.L2 rankCtx. tokens
Full Context
Claude Sonnet 51D0.7390.5700.5040.4600.4290.3520.62820.4730.4260.3710.428352,435
DeepSeek-v4-flash1D0.7660.5260.5720.4720.4550.3530.66910.5100.4780.3740.469152,435
Qwen-3.5-9b1D0.6220.4670.3440.3410.2850.2780.494120.4460.3430.2260.348652,435
GPT-5.6-terra1D0.6850.6230.4360.4530.3770.3560.57040.5050.4350.3030.431252,435
Retrieval Augmented
BM251D0.6760.6120.3780.4040.3130.3060.53960.4460.3760.2300.370410,769
Dense retrieval1D0.6510.5950.3740.3970.3580.3510.52980.4280.3680.2420.36259,539
GraphRAG2D0.6180.5540.3560.3840.3560.3430.504110.3990.3510.2360.344718,468
RAPTOR3D0.6470.5810.3500.3760.3450.3290.51790.4220.3270.2370.334814,151
External Memory
Mem01D0.4640.4080.2520.2640.2340.2230.370140.3050.2410.1720.245111,190
LightMem1D0.2900.2610.1330.1550.1780.1890.227150.1920.1310.0940.13915742
A-MEM2D0.5370.4060.3420.3000.3550.2610.454130.3210.2180.1850.235125,387
MemoryOS2D0.6230.4510.3870.3270.3270.2340.513100.3800.2550.1680.26995,203
xMemory3D0.6200.5110.4550.3740.4350.2620.54650.1330.2230.2200.203145,841
EverMemOS3D0.6790.5550.5400.4300.5610.3020.62130.1370.2840.2530.2471021,123
SpeakerMem-R13D0.6160.4690.4300.3230.4190.2360.53470.2120.2250.2080.220137,178

SpeakerMem-R1 achieves L1 of 0.534 but L2 of 0.220, within the range of existing external memory systems. This result highlights a challenge beyond the scope of its original group memory evaluations on GroupMemBench, SocialMemBench, and EverMemBench. These benchmarks use synthetic conversations and evaluate memory recall through reference answers or predefined response options. Our benchmark instead uses conversations from a deployed group assistant and evaluates whether remembered preferences and current decisions support assistance without a fixed answer. L2 requires integrating evidence across members and time into a response that satisfies both memory constraints and service requirements, with penalties informed by authentic member feedback. The added comparison shows that this challenge persists even for a framework designed for group memory. Evaluating group memory therefore requires connecting recall to the quality of assistance in real conversations. We have added SpeakerMem-R1 to the baseline description in Section 4.1 and the main results in Table 3, and discussed its implications in Section 4.2.

[Q1] Exact count per cell and the number of distinct source chats contributing to each. The per-cell counts are in Table 2 (776 / 448 / 150 for L1-M1/M2/M3; 221 / 650 / 150 for L2). Every cell is large enough that the differences Table 3 is read for, between paradigms and across M1–M3, are statistically distinguishable. The number of source chats (groups) per cell, which we add to Table 2, is:

M1M2M3
L1: groups (share of the 5 largest)50 (18%)50 (18%)46 (26%)
L2: groups (share of the 5 largest)44 (38%)46 (20%)43 (37%)

Every cell draws on at least 43 of the 50 groups, and no single group exceeds 10% of a cell. The distribution does not drive Table 3: weighting groups equally changes ranks by at most two positions on L1 and one on L2 (Spearman ρ = 0.98 / 0.99), dropping any single group leaves 12 of the 14 ranks unchanged on both levels, and in every case the three paradigm tiers are unchanged and only adjacent systems within a tier swap. We reword the claims that rested on those positions. We have expanded Table 2 in Section 3.2 to report the number of source groups contributing to each cell and the share of instances from its five largest groups.

[Q2] Confidence intervals accounting for dependence within chats. Thank you for this suggestion. We computed 95% percentile CIs from 2,000 bootstrap replicates. Each replicate samples 50 chats with replacement and retains their questions, using the same sampled chats for paired comparisons. The table reports overall scores, the ranks in Table 3, and the ranks by bootstrap mean with 95% rank intervals. We will report paired comparisons in the revision.

MethodL1 Avg. met [95% CI]L1 Table 3 rankL1 bootstrap mean rank [95% interval]L2 Avg. [95% CI]L2 Table 3 rankL2 bootstrap mean rank [95% interval]
Full Context
Claude Sonnet 50.628 [0.591, 0.664]22 [2–3]0.428 [0.387, 0.470]33 [2–3]
DeepSeek-v4-flash0.669 [0.634, 0.704]11 [1–1]0.469 [0.429, 0.509]11 [1–1]
Qwen-3.5-9b0.494 [0.461, 0.529]1212 [10–12]0.348 [0.315, 0.383]66 [5–8]
GPT-5.6-terra0.570 [0.538, 0.604]44 [4–5]0.431 [0.394, 0.464]22 [2–3]
Retrieval Augmented
BM250.539 [0.505, 0.573]66 [5–8]0.370 [0.336, 0.402]44 [4–5]
Dense retrieval0.529 [0.488, 0.570]88 [5–10]0.362 [0.330, 0.394]55 [4–6]
GraphRAG0.504 [0.466, 0.542]1111 [8–12]0.344 [0.306, 0.384]77 [5–8]
RAPTOR0.517 [0.481, 0.553]99 [7–11]0.334 [0.301, 0.369]88 [6–8]
External Memory
Mem00.370 [0.326, 0.414]1414 [14–14]0.245 [0.218, 0.272]1111 [10–12]
LightMem0.226 [0.200, 0.253]1515 [15–15]0.139 [0.123, 0.156]1515 [15–15]
A-MEM0.454 [0.414, 0.496]1313 [13–13]0.235 [0.211, 0.263]1212 [10–12]
MemoryOS0.513 [0.475, 0.551]1010 [7–12]0.269 [0.241, 0.298]99 [9–10]
xMemory0.546 [0.501, 0.591]55 [4–8]0.203 [0.176, 0.231]1414 [14–14]
EverMemOS0.620 [0.568, 0.669]33 [2–3]0.249 [0.217, 0.281]1010 [9–12]
SpeakerMem-R10.534 [0.495, 0.572]77 [5–10]0.220 [0.194, 0.248]1313 [12–13]

Ranking methods by their bootstrap mean reproduces the Table 3 ordering exactly on both L1 and L2. Individual replicates vary, as reflected in the rank intervals, but remain highly consistent with Table 3. The median Kendall’s τ between each replicate ranking and the Table 3 ranking is 0.92 on L1 [95% interval: 0.85, 0.98] and 0.96 on L2 [0.91, 1.00]. This variation mainly involves methods with close scores, which can exchange nearby ranks as resampling changes the contribution of individual chats. On L2, the rank intervals of RAG and external memory systems do not overlap. We have added the confidence intervals and ranking stability analysis to Appendix F.5, with a corresponding reference in the overall results discussion in Section 4.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, ACs, SACs, and PCs,

We appreciate your time and effort in reviewing our submission and are encouraged that reviewers highlight:

  • Problem formulation: the work shifts memory evaluation from storing the right fact to acting on it correctly for the right person (DvV7), and is "well motivated" with "strong practical relevance" (15eA).
  • Real-world data: the benchmark is built from records of a deployed group assistant with native assistance requests (DvV7, H2Lh, 15eA), "rather than synthetic templates" (DvV7).
  • Diagnostic design and findings: the L1/L2 split "cleanly separates retrieval quality from constraint application" (DvV7), showing that strong recall does not imply effective memory use (DvV7, H2Lh, 15eA).
  • Extensive evaluation: 14 configurations spanning full-context, RAG, and memory systems are evaluated (H2Lh, 15eA), and RER and ACB show "tangible improvements" (15eA).

We summarize our additional analyses and clarifications below. Revisions are highlighted in blue for DvV7, purple for H2Lh, and orange for 15eA.

DvV7 [W1] and H2Lh [Q3]: Evidence recovery and application. We add Oracle evidence, random reordering, and explicit instruction controls, together with a human answerability check. The results point to missing evidence and failure to apply available evidence as two sources of L2 failures. RER recovers missing evidence in flat memory and reorganizes available evidence in hierarchical memory. Related updates appear in Section 4.2 (Main Experiment), Section 5 (Analysis), and Appendices C.5, F.3, and F.4.

H2Lh [W1/Q1]: Memory use. We clarify that grounding criteria are essential in group assistance because members must see which statements a response relies on. Related updates appear in Section 3.3 (Evaluation) and Appendix C.4.

H2Lh [W2/Q2]: Contribution relative to related benchmarks. We compare evaluation and diagnostic designs with StreamMemBench and ContextWeave, clarifying our contribution in real group assistance, memory dependency types, and criteria informed by member feedback. Related updates appear in Section 1 (Introduction), Section 2 (Related Work), and Table 1.

DvV7 [W2/Q3]: Group memory baselines and configuration. We evaluate SpeakerMem-R1, showing that assistance remains challenging even for a framework designed for group memory. We also clarify shared stores, speaker attribution, and M1 as one required memory rather than one user. Related updates appear in Section 4.1 (Experimental Setup), Section 4.2, Table 3, and Appendix E.2.

DvV7 [Q1/Q2]: Sampling and statistical uncertainty. We add source group statistics and bootstrap confidence intervals by resampling chats. The bootstrap mean rankings remain consistent with Table 3. Related updates appear in Section 3.2 (Benchmark Construction), Table 2, Section 4.2, and Appendix F.5.

15eA [W1/Q1]: Completeness. We add a human audit of whether the rubrics cover relevant facts and member perspectives. Adding the few missed facts to the rubrics leaves system comparisons essentially unchanged, and no instance omits a member whose statements matter. Related updates appear in Section 4.1 and Appendix C.6.

15eA [W2/Q2]: Different members' needs. We provide a budget example accommodating simultaneously valid preferences and analyze 175 existing requests involving multiple members, making this task coverage explicit. Related updates appear in Section 3.1 (Task Formulation) and Appendix F.6.

While we have not yet received follow up comments, we would be glad to discuss any further questions during the remaining 24 hours of the discussion period. Thank you again for considering our responses and revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.