- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GroupAssistBench, a benchmark constructed from real-world group chats. The authors organize the benchmark along two dimensions, memory recall (L1) and memory use (L2), and three types of memory: single, distributed, and conflicting memories (M1–M3). Beyond traditional fixed-answer tasks that evaluate the retrieval capabilities of memory systems, the benchmark also emphasizes the practical value of memory use. Specifically, it includes tasks that assess how systems leverage conflicting and updated information in multi-party conversations to answer open-ended questions, such as planning a trip. The paper is well motivated and addresses a problem with clear practical relevance. The authors conduct extensive experiments on existing approaches, including full-context, RAG-based, and external memory systems. The results reveal that current memory systems consistently struggle with memory use, providing valuable insights and directions for future research. Furthermore, based on the limitations identified through their experiments, the authors propose two intervention strategies, RER and ACB, both of which demonstrate effective improvements over the existing baselines.
Critical strengths
-
The paper is well motivated and has strong practical relevance. The authors focus on group conversations and specifically design tasks to evaluate memory use in open-ended scenarios, which is particularly valuable for real-world agent applications. Moreover, the evaluation dataset is constructed from group chat records collected in real-world deployment settings, making the benchmark more representative of practical human-AI interaction scenarios.
-
The authors evaluate several existing approaches, including full-context, RAG-based, and external memory systems. Their results reveal that even when existing memory systems perform well on memory retrieval, they still exhibit significant limitations in memory use. The authors further analyze the underlying reasons for this gap and propose two targeted intervention methods, RER and ACB, both of which demonstrate tangible improvements over the baselines.
Critical weaknesses
-
The authors conduct a human evaluation in Appendix C, but they acknowledge that completeness is not effectively assessed. For group-oriented tasks, incorporating the perspectives and preferences of all group members is also an important factor, which may affect the evaluation of memory systems' effectiveness.
-
The paper evaluates the ability to handle conflicting information through M3. However, according to Appendix A, its construction primarily focuses on cases where information has been historically updated and the latest state has already been confirmed. In real-world scenarios, group conversations may also involve differing preferences and requirements among multiple members, raising the open-ended challenge of how an agent can leverage memory to create plans that accommodate the needs of different group members. I believe incorporating such tasks would provide a more comprehensive assessment of the challenges associated with memory use in real-world group assistance.
Questions for authors
-
Regarding Weakness 1, the authors could conduct additional experiments to further validate the evaluation of completeness in the benchmark.
-
Regarding Weakness 2, the authors could consider incorporating the aforementioned scenario into the benchmark.
5 comments
[W2 & Q2] Accommodating different members' preferences and requirements. Thank you for highlighting this aspect. L2 M2 and M3 accommodate different members' needs through planning, resource allocation, recommendations, and communication, not only historical updates. Figure 1 combines the revised April 30 departure, extra ¥50 per person, a window-seat preference, and costs for nine travellers. The L2 M3 case in Appendix G combines a three-minute duration limit with another member's concern about video completion rate. The existing L2 M2 instance below illustrates simultaneously valid preferences.
L2 M2: A travel budget accommodating different priorities
Request: “Suggest a compromise for our travel budget that meets both sides' needs as far as possible.”
Memory anchors (A–F denote members):
- A1. A proposes a basic budget of ¥2,200 per person for meals and accommodation.
- A2. B prefers ¥2,500 for unusual activities and spontaneous additions.
- A3. C also prefers ¥2,500 and does not want both photography and dining reduced.
- A4. D requires shared and personal expenses to be distinguished.
- A5. A rejects sharing personal shopping costs by default.
- A6. E warns that averaging to ¥2,350 may satisfy neither side.
- A7. B considers the contingency reserve inadequate for nine travellers.
- A8. F suggests a basic package with optional upgrades rather than one budget for everyone.
Rubric (translated and abridged):
- G1, +3. Recall concerns about a numerical compromise and the inadequate reserve.
- S1, +1. Propose a shared basic budget for essentials and voluntary upgrades, explaining how both sides' needs are accommodated.
- G2, +1. Recall F's basic package and A's rule against sharing personal shopping costs by default.
- S2, +2. Specify a usable reserve limit, spending conditions, and approval rules for extra activities.
- G3, +2. Recall C's photography and dining requirements and D's distinction between shared and personal expenses.
- S3, +2. Separate shared costs, optional upgrades, and personal purchases in the payment plan.
- P1, −2. Penalize misremembering the original ¥2,200 basic budget.
- P2, −2. Penalize changes that violate the advance communication agreement.
The response must turn different members' priorities into an executable budget and payment arrangement, not merely summarize their opinions.
Evaluation on requests involving multiple members. Reviewing all evaluated L2 M2 and M3 item cards identified 175 requests across 36 chats (124 M2, 51 M3), or 17.2% of evaluated L2. The tables report request types, subset scores, and rankings.
| Request type | How members' needs or positions are addressed | Requests | M2 | M3 |
|---|---|---|---|---|
| Shared planning and allocation | Develop a common plan or allocate resources to accommodate different members' needs | 78 | 71 | 7 |
| Joint communication | Address multiple members' different states, concerns, or requirements in one response | 49 | 45 | 4 |
| Applying or arbitrating group disagreements | Apply agreed outcomes or reconcile differing accounts from multiple members | 48 | 8 | 40 |
| Total | 175 | 124 | 51 |
| Method | All L2 Avg. | Subset M2 (124) | Subset M3 (51) | Subset Avg. (175) | Subset rank |
|---|---|---|---|---|---|
| Full Context | |||||
| Claude Sonnet 5 | 0.428 | 0.359 | 0.458 | 0.388 | 2 |
| DeepSeek-v4-flash | 0.469 | 0.409 | 0.523 | 0.442 | 1 |
| Qwen-3.5-9b | 0.348 | 0.286 | 0.231 | 0.270 | 6 |
| GPT-5.6-terra | 0.431 | 0.362 | 0.269 | 0.335 | 3 |
| Retrieval Augmented | |||||
| BM25 | 0.370 | 0.327 | 0.196 | 0.289 | 4 |
| Dense retrieval | 0.362 | 0.278 | 0.260 | 0.273 | 5 |
| GraphRAG | 0.344 | 0.258 | 0.226 | 0.249 | 8 |
| RAPTOR | 0.334 | 0.275 | 0.216 | 0.258 | 7 |
| External Memory | |||||
| Mem0 | 0.245 | 0.167 | 0.184 | 0.172 | 12 |
| LightMem | 0.139 | 0.099 | 0.104 | 0.100 | 14 |
| A-MEM | 0.235 | 0.154 | 0.177 | 0.161 | 13 |
| MemoryOS | 0.269 | 0.212 | 0.174 | 0.201 | 10 |
| xMemory | 0.203 | 0.177 | 0.247 | 0.197 | 11 |
| EverMemOS | 0.249 | 0.208 | 0.282 | 0.230 | 9 |
All 14 systems score lower on this subset than on all L2. Rankings remain close to Table 3 (Kendall correlation 0.89, maximum shift two places). On M3, EverMemOS and xMemory score 0.282 and 0.247. EverMemOS's point estimate exceeds GPT-5.6-terra Full Context (0.269). This suggests potential for hierarchical organization to integrate members' evidence when resolving conflicts. The requests assess applying memory to differing member needs. We have clarified this coverage under the Memory axis in Section 3.1 (Task Formulation). Appendix F.6 (Assistance Involving Multiple Members) adds the planning example, request distribution, and subset results. These additions are highlighted in orange.
Dear Reviewer 15eA, thank you again for your valuable feedback. As the discussion period is nearing its end, we would appreciate hearing whether our responses have addressed your main concerns. If any part remains unclear or if you have any further questions, we would be happy to clarify anything further.
We thank Reviewer 15eA for the thoughtful feedback. Revisions addressing this review are highlighted in orange in the revised manuscript. We address each concern below.
[W1 & Q1] Completeness of the required memories, and whether the perspectives of all members are covered.
We would like to clarify that our construction pipeline first extracts candidate facts needed for the response from the history before the cutoff, then removes candidates whose absence would not change the response (Appendix A.3). Retained anchors are further checked for necessity during quality control (Appendix A.6). To complement these checks and the human validation in Appendix C, we conducted a supplementary audit of the coverage of relevant facts and member perspectives using the same three annotators and 60 L2 instances.
Method. Reading the whole history for every instance is impractical, so we used the 14 evaluated systems as the search: every message that any system's response explicitly cited and that is not an anchor of the instance became a candidate missed fact (668 candidates over 54 of the 60 instances; up to 10 per instance were adjudicated, giving priority to messages cited by several systems, 355 in all). For each candidate the annotators saw the request, the anchors, the positive criteria, and the candidate message with one message of context on either side, and judged whether it is a preference, constraint, fact, or decision that a good response should use and that the existing anchors and criteria do not cover. Pairwise agreement is 0.83–0.88. This search gives a lower bound on misses in general, but it is the right bound for the question at hand: a missed fact that no system uses cannot change any system's score, so the facts systems actually cited are exactly the misses that could affect the comparison.
Results. By majority vote, 19 of the 355 candidates (5.4%) are relevant facts that the rubric does not cover. They fall in 12 of the 60 instances (20%; 3 instances if unanimity is required) and concentrate in M3 (8 of 20 instances, against 1 of 20 for M1 and 3 of 20 for M2): M3 requests ask how a decision was reached step by step, so intermediate steps beyond the anchored decision points are arguably relevant. On member perspectives, 8 of the 19 are earlier replies of the assistant that the group adopted, and all 11 missed member statements come from members who are already represented among the anchors of that instance; no instance omits a member whose statements matter. Together with the construction statistic that 56% of M2 and 93% of M3 instances draw anchors from at least two members, the perspectives of the members involved are covered.
Effect on the evaluation. A fact missing from the rubric cannot be rewarded, so a response that uses it receives no credit and the automatic scores are lower bounds; since the candidates come mostly from the explicit citations of the full-context models, the under-crediting falls on the leading systems rather than favouring them. Adding the 19 facts to the rubrics as grounding criteria and crediting the systems that cited them changes the mean score of every system on the 60 instances by less than 0.02 and changes no rank, except a swap of the two leading full-context models that the clustered bootstrap already marks as inseparable (credit by citation is a proxy that favours the full-context models, which cite explicitly); even crediting every system for every missed fact could raise a mean score by at most 0.05, below the gaps between paradigms.