acceptodds
Under review as a conference paper at ICLR 2027

CoalMem: Evidence Coalition Selection for Multi-Party Conversational Memory

Abstract

Answering queries over long-term multi-party conversational memory often requires combining evidence distributed across participants and discussion phases. Existing memory retrieval approaches largely optimize individual message relevance, yet the resulting context may still omit critical evidence needed to answer the query. Meanwhile, generic diversity does not guarantee that different messages contribute complementary query-relevant information. We therefore formulate multi-party conversational memory retrieval as evidence coalition selection and propose CoalMem, a training-free framework for selecting a compact set of messages with high joint evidence value. It constructs an evidence coalition graph over seed retrieved messages, where query–evidence edges capture relevance and evidence–evidence edges characterize both non-redundancy and query-conditioned complementarity. It then performs query-anchored coalition optimization to jointly select a compact set of messages that are individually relevant and collectively supportive of the query, without additional LLM calls during selection. Experiments on GroupMemBench across six tasks show that our method consistently outperforms retrieval-based, memory-based, and alternative evidence-selection methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Party Conversational Memory, Evidence Coalition Optimization, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hKWEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of post-retrieval evidence selection in multi-party conversational memory. The authors observe that individually relevant messages do not necessarily form a collectively sufficient evidence set, and that generic diversity can only capture whether evidence items are different, but cannot capture whether the differences among them contribute complementary information. The authors formalize evidence selection as a query-anchored coalition optimization problem and propose CoALMem, a framework that constructs an evidence coalition graph containing query-evidence edges and evidence-evidence edges (encoding non-redundancy and query-conditioned complementarity), and then uses replicator dynamics to jointly estimate coalition member weights under a fixed evidence budget. It achieves sufficiently competitive performance on GroupMemBench and EverMemBench.

Critical strengths
  1. The insight found in this paper is somewhat innovative, and the overall presentation is relatively clear.
  2. CoalMem achieves sufficiently competitive performance on groupMemBench and EverMemBench.
Critical weaknesses

1.In multi-party conversations, information is highly dependent on context. The authors highlight "no LLM calls during selection" as a core point and rely solely on lexical BM25 to compute evidence non-redundancy and complementarity. However, BM25 struggles to capture deep semantic associations (e.g., coreference, synonym substitution, implicit logical complementarity), where LLMs have a natural advantage. The authors should therefore demonstrate that their algorithm can indeed outperform or match methods that directly introduce LLMs during the selection phase in terms of evidence selection quality (e.g., via specifically designed prompts for semantic-level redundancy elimination and complementary information filtering over the candidate pool).

2.In Table 2, using only Qwen3-Reranker-4B as the LLM-based reranker baseline is unconvincing. The authors need to include other strong cross-encoders, such as BGE-Reranker-v2-m3, or larger-parameter, Listwise strategy-based LLM Rerankers (e.g., RankGPT).

3.Lack of fine-grained efficiency analysis. Beyond the aggregated online latency reported in Appendix C.2, the authors need to provide a complete end-to-end time consumption breakdown, explicitly listing and comparing the specific time costs or token consumption across the three distinct stages: Retrieval, Selection (evidence selection/graph optimization computation), and Generation (final answer generation).

Questions for authors

Please see the weakness.

Reviewer_M6SgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes COALMEM for post-retrieval evidence selection in multi-party conversational memory. It first retrieves the top-N candidate messages with BM25 and constructs an evidence coalition graph. Query-message edges represent individual relevance, while message-message edges combine lexical non-redundancy and query-conditioned complementarity. The method then applies query-anchored quadratic optimization with replicator dynamics to assign coalition weights and selects the top-K messages as the final evidence set.

Critical strengths
  1. Evidence in multi-party conversations can be distributed across participants, channels, and discussion phases, so individual message relevance may not be sufficient.
  2. The interaction between query relevance, non-redundancy, complementarity, and subset optimization is clearly presented.
Critical weaknesses
  1. The main novelty is therefore the combination of these components for multi-party conversational memory.
  2. The proposed evidence complementarity is mainly defined through BM25 query-term coverage, while non-redundancy is derived from lexical overlap.
  3. The paper appears closer to a standard query-aware diversity reranker than a substantial contribution to conversational memory or evidence composition.
Questions for authors

I think that while this work performs well on the current dataset, it still feels like an engineering-driven framework and lacks novelty.

Reviewer_u61ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CoalMem proposes a training-free post-retrieval selection method for multi-party conversational memory QA. It builds a coalition graph over BM25 seed evidence, combining query-evidence relevance with pairwise utility from non-redundancy and query-conditioned complementarity, and selects a compact evidence set through query-anchored coalition optimization with replicator dynamics, with no extra LLM calls in selection.

Critical strengths
  1. Set-level selection is the right lens for multi-party memory, because evidence is split across participants and phases.
  2. The method is training free and needs no extra language model calls during selection, which is a practical contrast to memory baselines that need large construction costs.
  3. It beats the strongest baseline in its own comparison set, not just its seed retriever, with Hindsight and HippoRAG as the previous best on GroupMemBench and a neural reranker under matched pool conditions.
  4. Online latency is modest compared with the graph and memory baselines, and there are no memory construction calls.
  5. Abstention accuracy is improved or held alongside answer accuracy, which matters for a memory assistant that should not hallucinate from partial evidence.
Critical weaknesses
  1. Its state of the art claim is incomplete, because it does not compare against newer multi-party specific systems or against constrained dominant sets for evidence selection, so I cannot tell from the paper whether it still wins against current state of the art.
  2. Answers are judged by the same model that generated them, with no human or cross model judge and no variance or significance reported, and some categories are small, so I discount the headline gaps.
  3. Both benchmarks are synthetic group conversations, with no real multi-party data, no human evaluation, and no test of whether the gains survive noisy or incomplete real chat.
  4. Complementarity and non-redundancy are both BM25 based, so paraphrase and semantic complementarity are likely missed, and performance leans on clean speaker, role, channel, topic, and phase metadata, which may be noisy or missing in practice.
Questions for authors
  1. Can you report results with an independent judge, either a different model or a human check on a subset, plus variance and significance for the matched pool gaps?
  2. How do latency and cost change as N and history length grow, and how do you handle incremental updates when new messages arrive without rebuilding the graph from scratch?
  3. How would you extend complementarity beyond lexical term coverage to capture paraphrase or semantic complementarity, and would that help the categories where you still struggle, like knowledge update and term ambiguity?

Comments on the paper

1 comment

Thank you for the authors' response. However, I remain concerned about the limited methodological novelty of this work.

First, recent Agent Memory studies, such as MemGAS [1] and Mem0 [2], have already explored long-term conversational memory organization, association, retrieval, and selection. In comparison, CoalMem mainly combines established relevance scoring, non-redundancy modeling, and graph optimization techniques. Its proposed Evidence Complementarity is also based on BM25 query-term coverage, offering limited methodological novelty.

Second, the evaluation lacks direct comparisons with representative memory systems such as Mem0 and MemGAS, making it difficult to establish its advantages over existing memory approaches.

More importantly, I believe CoalMem is essentially an evidence reranking and selection method within a conventional RAG pipeline, with multi-party conversational memory serving primarily as a new application setting. The core algorithm does not introduce novel mechanisms for memory construction, updating, or memory-specific reasoning. Therefore, its contribution to this research field remains limited.

Although the empirical improvements are encouraging, the rebuttal primarily emphasizes the combination of existing techniques and performance gains without sufficiently addressing the fundamental novelty concerns. Therefore, I maintain my original score.

[1] From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents.

[2] Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.