acceptodds
Under review as a conference paper at ICLR 2027

IGMem: Interaction-Grounded Memory for Multimodal Multi-Party Conversations

Abstract

Agent memory for multimodal multi-party conversations aims to support recall and reasoning about past interactions and events, whose constituent information is often distributed across participants, turns, and modalities. Building such memory is challenging for two reasons: information about the same event can be scattered across different speakers, turns, and modalities, and different participants may attend to different parts of the same multimodal content. Existing approaches typically organize memory by temporal proximity, semantic similarity, or generic visual content, but overlook these interaction-dependent relationships, leaving event information fragmented or ignored. To address these challenges, we introduce IGMem, a novel Interaction-Grounded Memory framework for multimodal multi-party conversations. Our key insight is that each response provides a signal for connecting distributed sources and selecting the particular content relevant to it. Specifically, IGMem first recovers the interaction flow from multiple conversational cues, and then compiles the relevant textual and visual evidence into response-centered memory units. Such memory units brings information belonging to the same event together, making it easier to retrieve and reason over past information. Extensive experiments show that IGMem outperforms competitive baselines on two benchmarks across different backbones. Our code is available at https://anonymous.4open.science/r/IGMem.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agent, Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Lnd4OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies agent memory for multimodal multi-party conversations, where information about one event is spread across speakers, turns and images, and different participants pick up different details of the same content. IGMem links each turn to earlier turns using recency, same-speaker and global semantic cues, then has an LLM compile a short, turn-conditioned context from those sources, which is propagated recursively and stored with the original turn. On H2HMem and Mem-Gallery with two backbones, it improves the overall LLM-Judge score by 5.5 to 10.5 points over the strongest baseline.

Critical strengths

The problem is clearly motivated, and the method follows directly from it, with each component (flow recovery, conditioned uptake, recursive propagation) shown to matter in the ablations. The gains are consistent across two benchmarks and two backbones, results are averaged over three runs, and code is provided. The uptake-quality audit in Appendix C.2 is a useful addition that many memory papers lack.

Critical weaknesses
  1. Construction cost is not reported. IGMem calls a multimodal LLM once per turn (over 7,000 turns on H2HMem), while retrieval baselines such as MuRAG and NaiveRAG do no write-time processing. Figure 4 compares only query-time tokens, so the efficiency claim and the comparison with these baselines are incomplete.

  2. Answer Refusal drops in every setting (by 9.4 and 12.6 points on H2HMem, 4.3 and 6.0 on Mem-Gallery). Together with the 84.3% factual-grounding rate for GPT-4.1-nano in Appendix C.2, this suggests that compiled contexts introduce unsupported information that leads the model to answer when it should not, and recursive propagation may spread these errors.

  3. The core idea, enriching each memory entry with LLM-written context at write time, is close to A-Mem and contextual retrieval approaches. Appendix C.1 compares against a variant of IGMem's own pipeline rather than an existing enrichment method.

Questions for authors

My positive score depends on the following being addressed in the rebuttal:

  1. Please report memory-construction cost (LLM calls, tokens and time) for IGMem and all baselines, and add a baseline that applies comparable write-time LLM enrichment to MuRAG, so the gain from interaction grounding can be separated from the gain from extra LLM processing.

  2. Please analyze the drop in Answer Refusal. How often do refusal failures trace back to unsupported content in compiled contexts, and do grounding errors accumulate with recursion depth?

If these show that the gains hold under matched construction effort and that error propagation is limited, I will keep my score. Otherwise I would lower it.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Lnd4 (Part I)

Dear Reviewer Lnd4,

We sincerely thank you for recognizing our work and for your generous rating, which greatly encourages us. We also appreciate your questions and suggestions and hope our point-by-point responses address your concerns.

W1 & Q1: Memory-construction costs and comparable write-time enrichment for MuRAG

Thank you for raising this important issue. We reported construction costs for LLM-based methods in Appendix C.4 due to space limits. We will move this analysis into the main text. Below, we compare construction costs across methods and test MuRAG with a comparable write-time LLM budget.

Memory-construction Cost Analysis

Experimental setup and metrics. We use H2HMem with GLM-5.3-Flash and the construction settings from our main experiments. All methods build memory from scratch on the same 25 conversation histories. Experiments run on a server with an NVIDIA H20 GPU and a 64-core CPU (128GB RAM). LLM calls use OpenRouter; the local GPU handles embedding encoding and similarity computation.

For each history, we sum construction-time LLM calls and API-reported prompt_tokens and completion_tokens, then average across histories. Image captions for text-based methods are provided by H2HMem, so their generation costs are excluded, although they would incur costs in practice. We report LLM calls and tokens instead of wall-clock time, which depends on API load, network conditions, etc.

Experimental results. Values are averaged per history; M denotes million tokens.

MethodLLM CallsInput (M)Output (M)Total (M)
NaiveRAG————
MuRAG————
NGM————
A-Mem573.360.9751.7622.737
Mem0578.560.8540.4051.259
MemGPT353.922.1740.7082.882
MIRIX1,981.845.9461.1897.135
IGMem333.240.9550.0811.036

Result analysis. IGMem uses 333.24 calls and 1.036M total tokens, 17.7% fewer than Mem0 and 62.1%–85.5% fewer than A-Mem, MemGPT, and MIRIX. Its input cost (0.955M) is close to A-Mem (0.975M) and above Mem0 (0.854M). The main saving comes from output tokens (0.081M, 80.0%–95.4% fewer), as IGMem preserves original turns and generates only compact, response-relevant contexts.

NaiveRAG, MuRAG, and NGM need no construction-time generative LLM calls under our settings. IGMem adds a one-time construction cost to organize distributed evidence, but achieves better QA performance at comparable query-time token budgets (Section 4.4, Figure 4). This follows the common design of agent memory systems, which invest in memory construction to improve subsequent query efficiency and accuracy.

Comparison with LLM-enriched MuRAG

LLM-enriched MuRAG. We extend MuRAG with local-window context enrichment. For each turn, we provide an LLM with the current message and information of its preceding eight turns. The LLM generates an enriched message while preserving source-supported facts, speaker attribution, temporal conditions, and uncertainty. Enriched text is limited to twice the original length. The window size is chosen to maintain a construction cost comparable to IGMem. The enriched text replaces the original text and is re-encoded for retrieval. Query processing follows the original MuRAG pipeline, retrieving enriched records by similarity and supplying them to the answer model.

Experimental setup and results. We use H2HMem with GLM-5.3-Flash and the main experimental settings. Results are averaged over three runs.

MethodBuild Input (M)Build Output (M)Query Tokens / QuestionOverall
MuRAG (Top-6)——2,1670.7034
LLM-enriched MuRAG (Top-6)0.9130.1272,6490.6811
IGMem (Top-5)0.9550.0812,0950.7605

Result analysis. LLM-enriched MuRAG uses 1.040M construction tokens versus IGMem's 1.036M (0.4% difference). With Top-6 retrieval, enrichment lowers MuRAG's Overall from 0.7034 to 0.6811 (−2.23 pp) and raises query tokens from 2,167 to 2,649 (+22.2%). IGMem reaches 0.7605 (7.94 pp higher) using 2,095 query tokens (20.9% fewer).

This suggests that additional LLM processing and longer contexts do not necessarily improve QA performance. Local-window enrichment may introduce irrelevant information from interleaved conversations or omit important details when rewriting original records. IGMem instead recovers antecedents through temporal, participant, and semantic cues, and compiles response-relevant context while preserving original observations. This allows relevant evidence to remain jointly accessible for downstream retrieval and reasoning.

AuthorsOpenReview ↗

Response to Reviewer Lnd4 (Part II)

W2 & Q2: Analyze the drop in Answer Refusal. How often do refusal failures trace back to unsupported content in compiled contexts, and do grounding errors accumulate with recursion depth?

Thank you for your valuable suggestion. We agree that further analysis of the Answer Refusal (AR) performance drop would help clarify the limitations of IGMem.

In Appendix C.2, we briefly evaluated the quality of compiled contexts. We will expand this section with more detailed analysis and discussion in the revised paper.

In the following, we first analyze AR failures and then examine the sources and propagation of errors in compiled contexts.

Answer Refusal Analysis

Error categorization. To better understand the failures on Answer Refusal (AR), we classify the failure cases into four categories:

  1. Content hallucination: The compiled context introduces unsupported information that is not present in the original evidence, leading to an incorrect answer.
  2. Detail omission: The compiled context omits important details from the original evidence, affecting the final answer.
  3. LLM hallucination: The answer model generates an incorrect answer despite receiving a context without identified errors.
  4. Annotation ambiguity: The question is ambiguous, making it difficult to determine whether the model should refuse to answer.

Error distribution. We analyze the AR failure cases on H2HMem with GLM-5.3-Flash. We compare IGMem with two representative baselines: Full (Text), which achieves the highest AR score among baselines, and MuRAG, which achieves the highest Overall score among baselines. The distribution of failure types is shown below.

MethodOverallAR FailuresContent HallucinationDetail OmissionLLM HallucinationAnnotation Ambiguity
Full (Text)0.44872400213
MuRAG0.69005200475
IGMem0.76055822495

Results and discussion. Among the 58 AR failures of IGMem, content hallucination and detail omission each account for only 2 cases (3.4%). Both types occur during the compilation of textual and visual evidence.

For content hallucination, we find cases where the LLM incorrectly interprets visual information when integrating it with textual evidence. For example, a user mentioned leaving a positive review for a store, while an attached image showed the store's overall rating. During compilation, the LLM incorrectly attributed this rating to the user. For detail omission, we observe cases where the LLM overlooks time-related restrictions shown in images, leading to unsupported inferences during answer generation.

In contrast, 49 failures (84.5%) are attributed to LLM hallucination during answer generation, suggesting that memory construction errors have a limited impact on AR performance. Similar patterns appear in Full (Text) (87.5%) and MuRAG (90.4%), indicating that this issue is not specific to IGMem.

Although IGMem improves the Overall score from MuRAG's 0.6900 to 0.7605, its AR performance remains lower. The main results show a similar trend: methods with higher Overall scores often achieve lower AR scores. One possible explanation is that more coherent and compact contexts encourage the answer model to infer details beyond the available evidence.

Error Accumulation in IGMem

Compiled-context quality. In Appendix C.2, we evaluate 1,000 randomly sampled turns from H2HMem using GPT-4.1-nano and GLM-5.3-Flash. GPT-6 Astra assesses each compiled context against the original textual and visual evidence, including sources inherited through intermediate turns, followed by human verification. We examine three criteria: (1) factual grounding, whether claims are supported by original evidence; (2) attribution and qualification, whether speaker attribution and factual qualifications are preserved; and (3) relevant-evidence coverage, whether response-relevant information is retained. With GLM-5.3-Flash, the pass rates are 92.6%, 96.1%, and 81.6%, respectively, indicating generally reliable compilation but remaining errors and omissions.

Error attribution and propagation. We further examine the 74 cases that fail factual grounding with GLM-5.3-Flash. Among them, 69 involve direct compilation errors, where the LLM introduces unsupported information from original evidence, while only 5 involve inherited errors from antecedent contexts. We also trace error propagation to subsequent turns. These 74 turns serve as antecedents for 192 later turns, but only 15 (7.8%) inherit the incorrect information.

Results and discussion. These results suggest that most grounding errors arise from direct compilation rather than recursive error accumulation. Error propagation is also limited, as turn-conditioned uptake compilation selectively retains response-relevant information, potentially filtering out incorrect details inherited from antecedents.

AuthorsOpenReview ↗

Response to Reviewer Lnd4 (Part III)

W3: Comparison between our core idea and A-Mem and other contextual retrieval approaches

Thank you for your question. In Section 4.2, we compare IGMem with A-Mem and other contextual memory methods. However, the key design differences may not have been sufficiently explained due to space limitations. We will clarify these differences in the revised paper.

Comparison with Existing Methods

Enriching memory entries with LLM-generated context is a common goal of memory systems, but existing methods differ in how they achieve it.

A-Mem. A-Mem uses LLMs to generate keywords, tags, and contextual descriptions for memory notes. It retrieves semantically related notes and uses LLMs to link and update them. However, semantic similarity may miss interleaved dependencies, while note linking does not explicitly capture individual uptake. IGMem instead combines multiple cues to recover antecedents and selects source details conditioned on the current response.

Mem0. Mem0 extracts important facts from recent messages and conversation summaries. It retrieves related memories and uses LLMs to add, update, delete, or retain facts. However, this process may miss distant dependencies and lose the connection between responses and their specific sources. IGMem instead recovers cross-turn antecedents and preserves response-specific evidence through recursive context propagation.

MemGPT. MemGPT manages information through hierarchical context and external storage. It recursively summarizes older messages and retrieves historical records when needed. However, hierarchical storage and summarization do not explicitly address interleaved dependencies or individual uptake. IGMem instead organizes distributed evidence around each response and propagates relevant context across turns.

MIRIX. MIRIX organizes multimodal information into six memory types using specialized managers. A Meta Memory Manager coordinates memory storage and retrieval across these types. However, this organization does not explicitly capture cross-participant dependencies or how different responses use the same source. IGMem instead recovers response-relevant antecedents and selectively compiles the textual and visual details taken up by each response.

IGMem. The key difference lies in how contextual information is selected and propagated. IGMem combines three mechanisms: multi-cue antecedent selection, turn-conditioned uptake compilation, and recursive context propagation. Together, they organize distributed interaction evidence and response-specific details into retrievable memory units.

Overall Performance Comparison

Tables 1 and 2 show that IGMem consistently outperforms A-Mem and other memory baselines in Overall scores across both benchmarks and backbones.

On H2HMem, IGMem improves Overall scores over A-Mem from 0.5757 to 0.6841 (+10.84 percentage points) with GPT-4.1-nano, and from 0.5965 to 0.7605 (+16.40 points) with GLM-5.3-Flash. On Mem-Gallery, the corresponding improvements are from 0.7344 to 0.8042 (+6.98 points) and from 0.8214 to 0.8939 (+7.25 points).

These results support the benefit of interaction-grounded memory construction beyond existing contextual enrichment methods. The ablation studies in Section 4.3 further demonstrate the contributions of each component.

Reviewer_QAGROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies memory construction for multi-party conversations. The paper introduces IGMem and shows its strong performance on two benchmarks.

Critical strengths
  1. The paper propose an interaction-based memory construction, IGMem, for multimodal multi-party conversation.

  2. The experiments show IGMem consistently outperforms strong memory baselines by up to 10.45%.

Critical weaknesses
  1. The challenges of the task are not clear. Interleaved dependencies and indivisual uptakes are the propertie of conversations instead of challenges. It is not clear why the baselines' performance are not good since they should be able to capture these properties, like by retrieval.

  2. The terms in the paper are confusing. For examples, what does a "response" mean? Do you mean a "message" that migh be a question, answer or statement; from my understanding, "relation" is more accurate than "interaction" used in the paper, since the IGMem mainly consider the relation between the messages; what is "A" in equation 1 and "F" in equation 12?

  3. The technique to obtain the relation is not sound. The author selects the top k semantically relevant turns for memory unit by computing the cosine similarity between the current and historical turn representations. It is not clear this computation is valid since two messages might not be similar even they are strongly related like question and answer.

Questions for authors

In the experiment,

a) what are the baseline settings and the difference between IGMem;

b) what is specific LLM for the judge?

c) what are the sota performances for the benchmark?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QAGR (Part I)

Dear Reviewer QAGR,

We sincerely thank you for your recognition of IGMem and its experimental results. We also appreciate your valuable comments on the clarity of our task motivation, terminology, and experimental setup. We hope our point-by-point responses will clarify these issues and address your concerns.

W1: The challenges of the task are not clear.

Thank you for your suggestion. We provide a clearer explanation below. First, we clarify the challenges introduced by interleaved dependencies and individual uptake. Next, we explain why retrieving relevant records may still leave incomplete evidence. Finally, we use ablation results to examine how IGMem addresses these challenges.

Challenges Introduced by Interleaved Dependencies and Individual Uptake

Interleaved dependencies. A single message may lack the context needed to understand its references or decisions. Enriching each memory entry with relevant context is therefore a common goal of agent memory systems.

However, in multi-party conversations, a response may depend on information from different participants, nonadjacent turns, and multiple modalities. Meanwhile, adjacent messages may belong to unrelated topics. Memory construction must identify and connect the sources relevant to each response. Semantic similarity alone or fixed temporal windows may miss distant sources or introduce unrelated information.

For example, in Figure 1, Chloe's final dinner proposal combines Restaurant B from Alice's second image, 7:30 from the later scheduling exchange, and Saturday from the initial plan for Restaurant A. Understanding this proposal requires connecting these distributed sources. Neither the final message alone nor a single restaurant-related record contains all the evidence.

Individual uptake. Different participants may focus on different details of the same multimodal source. In Figure 1, Ben focuses on the signature chicken in Restaurant A's menu, while Chloe notices the 20% service fee after 6 pm. Although both respond to the same image, they take up different information. Generic LLM-generated captions or source-level multimodal embeddings cannot ensure that these response-specific details are preserved for later retrieval.

Why Retrieval Alone May Still Leave Incomplete Evidence

We agree that retrieval can recover some of these connections. However, retrieving relevant records does not necessarily provide complete evidence for answering a question. Similarity-based retrieval mainly identifies records relevant to the query, while answering may require information distributed across multiple turns. Under a limited Top-K budget, indirect sources containing key conditions may not rank highly. Moreover, individual records may lack their dependent context, while generic image descriptions may omit important visual details.

For example, retrieving Restaurant B's image alone cannot recover the Saturday date and 7:30 time from other turns. Similarly, retrieving Restaurant A's menu without the 20% service fee detail cannot fully explain Chloe's concern. In both cases, relevant records are retrieved, but the evidence remains incomplete.

This motivates contextual enrichment in existing memory systems, which aims to improve memory representations and evidence coverage before retrieval. IGMem follows this direction but organizes memory around conversational interactions. It connects distributed antecedents, compiles response-relevant textual and visual details, and recursively propagates relevant context. This makes related records and their supporting evidence jointly accessible for downstream retrieval and reasoning.

Experimental Evidence for the Proposed Designs

Experimental setup. We use the ablation study on H2HMem with GLM-5.3-Flash (Section 4.3). We separately replace Conversation Flow Recovery and turn-conditioned compilation, or remove recursive propagation, while keeping the remaining settings fixed.

Experimental results.

MethodOverallDrop (pp)
IGMem0.7605—
w/o Flow Recovery0.6746−8.59
w/o Conditioned Uptake0.6831−7.74
w/o Recursive Propagation0.7122−4.83

Result analysis. Replacing flow recovery with chronological connections reduces Overall from 0.7605 to 0.6746, showing the importance of selecting relevant antecedents. Replacing turn-conditioned compilation with generic summaries reduces Overall to 0.6831, supporting response-specific content selection. Removing recursive propagation reduces Overall to 0.7122, showing the benefit of retaining context from earlier interactions.

Together, these results support the contributions of antecedent selection, turn-conditioned uptake, and recursive context propagation to downstream QA performance.

AuthorsOpenReview ↗

Response to Reviewer QAGR (Part II)

W2: The terms in the paper are confusing.

Thank you for pointing this out. We apologize for the unclear terminology and notation in the original paper. We will clarify these terms upon their first use and provide explicit definitions of the functions and variables in the revision.

What does a response mean?

Your understanding is correct. A response refers to a message from a participant in the conversation. It can be a question, an answer, a statement, or a message with an image. It does not have to be a direct answer to a question.

We used response to emphasize the current message's position in the conversation and its connection to earlier messages. However, we agree that this term may be confused with an answer. Using message or conversational turn would be clearer.

Why do we use interaction rather than relation?

We use interaction to emphasize how participants exchange information through messages and how the current message draws on earlier contributions.

The term relation is broader and may include general associations such as topic similarity, entity co-occurrence, or temporal proximity. Our focus is specifically on dependencies formed through conversational interactions. We agree that interaction relation may better clarify this distinction.

Definitions of F_build, F_answer, and A

We apologize for not clearly defining these symbols near the equations. Below, we clarify the main notation used in our paper.

SymbolDefinition
HHConversation history
MMMemory constructed from the conversation history
FbuildF_{\mathrm{build}}Memory construction process
FanswerF_{\mathrm{answer}}Process of answering a query using memory
AAMultimodal answer model that generates an answer from the query and retrieved context
qqEvaluation query
CqC_qRetrieved context provided to the answer model for query qq
y^\hat{y}Answer generated by the model

W3: The technique to obtain the relation is not sound.

Thank you for pointing this out. We agree that strong interaction dependencies do not necessarily imply high embedding similarity. Relying only on cosine similarity may therefore miss relevant antecedents. IGMem addresses this by combining three complementary cues for antecedent selection. We have compared different selection strategies in Appendix C.1 and will highlight these results in the revised paper.

Antecedent Selection and Compilation

IGMem combines three cues to recover candidate antecedents:

  • Temporal continuity. IGMem always retains the immediate predecessor, regardless of semantic similarity, and selects additional candidates from a recent window. For example, "What time should we meet?" followed by "7:30" can be connected even with low embedding similarity.
  • Participant-grounded reference. IGMem selects relevant turns from the same speaker's history. This prevents earlier contributions from being overlooked when competing with all historical turns.
  • Cross-modal semantic affinity. IGMem uses a shared multimodal encoder to retrieve relevant sources across the entire history. This captures distant textual and visual information, although semantic similarity alone cannot guarantee actual dependencies.

After selection, Recursive Individual Uptake Compilation uses the current turn to select relevant information from antecedents' original observations and accumulated contexts. This helps filter irrelevant content while preserving source-supported details and attribution. However, compilation cannot recover sources missed during selection.

Controlled Comparison of Antecedent Selectors

Experimental setup. In Appendix C.1, we compare four selection strategies on H2HMem with GLM-5.3-Flash. We fix the encoder, compiler, recursive propagation, context length, and query processing. Only antecedent selection varies, with the same budget B = 6.

Antecedent SelectionOverallIGMem Gain
Recent-B0.6746+8.59
Global Semantic0.7231+3.74
Same-Speaker Semantic0.6833+7.72
IGMem (Three Cues)0.7605—

Results and discussion. Global Semantic outperforms Recent-B and Same-Speaker Semantic, showing the value of global semantic retrieval. IGMem further improves Overall by 3.74 percentage points, suggesting that temporal and participant cues provide complementary information. Compared with Recent-B and Same-Speaker Semantic, the gains reach 8.59 and 7.72 points, respectively.

These results support combining multiple cues rather than relying on semantic similarity alone. However, our method still uses heuristic selection and a bounded antecedent budget. It may miss sources with low similarity but strong interaction dependencies. More accurate recovery of these dependencies remains an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer QAGR (Part III)

Q1: What are the baseline settings and the differences between IGMem and the baselines?

Thank you for your suggestion. We provide baseline configurations in Appendix B, but the key design differences may not have been sufficiently explained due to space limitations. We will clarify these differences in the revised paper.

Experimental Settings

We follow the H2HMem benchmark adaptations for Full (Text), NaiveRAG, A-Mem, Full (MM), MuRAG, and NGM. For Mem0, MemGPT, and MIRIX, we apply the same modality-handling principles. All baselines use their recommended settings.

We evaluate GPT-4.1-nano and GLM-5.3-Flash as backbones, using the same LLM for construction and answering when applicable. We use GME-Qwen2-VL-7B-Instruct as the shared multimodal encoder, Top-5 retrieval for retrieval-based methods, and average results over 3 runs.

Difference with Baselines

  • Full (Text). It directly provides historical text and image captions within the context limit. Unlike IGMem, it requires the answer model to recover interleaved dependencies and individual uptake from raw history, which may also be truncated.

  • NaiveRAG. It retrieves text chunks and image captions by semantic similarity. Unlike IGMem, it does not connect distributed sources into response-centered memory units, potentially leaving event information fragmented.

  • A-Mem. It enriches and links memory notes mainly through semantic similarity and LLM-based updates. Unlike IGMem, it does not explicitly recover interaction dependencies or select source details according to individual uptake.

  • Mem0. It extracts and updates facts using recent messages and conversation summaries. Unlike IGMem, it does not explicitly preserve cross-turn dependencies or which source details are taken up by each response.

  • Full (MM). It directly provides original text and images within the context limit. Unlike IGMem, it leaves the recovery of distributed sources and response-specific visual details to the answer model.

  • MuRAG. It retrieves original multimodal records based on query similarity. Unlike IGMem, it does not compile indirect dependencies or response-specific visual details into the retrieved memory units.

  • NGM. It incrementally maintains a compressed representation of conversation history. Unlike IGMem, it does not explicitly preserve response-specific source dependencies and individual uptake during compression.

  • MemGPT. It manages memory through hierarchical storage, recursive summarization, and historical retrieval. Unlike IGMem, its recursive summaries do not explicitly organize interleaved dependencies and individual uptake around each response.

  • MIRIX. It organizes multimodal information into six memory types through specialized managers. Unlike IGMem, it does not explicitly connect cross-participant contributions to the responses that use them or distinguish their individual uptake.

Key difference. Existing methods organize memories through retrieval, summarization, fact updates, or structured storage. IGMem instead focuses on interaction-grounded memory construction. It combines multi-cue antecedent selection, turn-conditioned uptake compilation, and recursive context propagation to preserve distributed evidence and response-specific details.

Additional Key Difference for Multimodal information

For multimodal information, existing methods mainly rely on raw image representations or general captions. Neither explicitly captures how different participants take up different details of the same image. IGMem instead selects visual details conditioned on the current response and preserves them alongside the original observations for downstream retrieval and reasoning.

A common limitation of these baselines is the lack of explicit modeling of individual uptake. Different participants may attend to different details of the same image. Generic captions may omit these details, while image embeddings may not emphasize their relevance to a particular response.

IGMem instead uses turn-conditioned uptake compilation to select relevant visual details from original images and textual evidence. It preserves these details in the compiled context while retaining the original observations. This connects multimodal information to specific interactions, making response-relevant evidence more accessible for downstream retrieval and reasoning.

Reviewer_VYRmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies memory for multimodal conversations involving multiple people. Its main idea is to build memory around each response: retrieve earlier turns that may have informed the response, then summarize the parts of those turns that matter for understanding it. The resulting memory units are used for later retrieval and QA. Results on H2HMem and Mem-Gallery show consistent gains in overall score.

Critical strengths

A sensible problem formulation

The paper focuses on a real limitation of turn-based memory: in a group conversation, one reply can depend on details scattered across earlier speakers, turns, and images. Using the response itself to decide what to retain is a reasonable and useful design choice.

Good end-to-end evidence

The empirical section is thorough. IGMem improves overall results on two benchmarks and two backbones, and the ablations support the usefulness of flow recovery, conditioned uptake, and recursive propagation. The token and construction-cost analysis is also helpful.

Critical weaknesses

The quality of recovered interaction flow is not directly assessed

IGMem selects candidate antecedents using temporal, speaker, and semantic cues. The ablations show that this strategy improves downstream QA, but it remains unclear how accurately these edges reflect actual conversational dependencies. A small human audit of selected antecedents, or another direct flow-quality metric, would make the interaction-grounded interpretation more convincing and clarify whether the gains come from better dependency recovery rather than generic context enrichment.

Answer refusal consistently regresses

IGMem underperforms the strongest baseline on Answer Refusal in both benchmarks and with both backbones, with particularly large drops on H2HMem. This is important for a persistent memory system, which should avoid answering when the available history is insufficient. The paper would benefit from an error analysis or representative examples to clarify whether the issue comes from retrieval, context compilation, or recursive propagation, and from discussing possible safeguards for this trade-off.

Questions for authors
  1. Could the authors directly validate the recovered flow on a small sample, for example by asking annotators whether the selected antecedents genuinely support the current response?

  2. Why does IGMem consistently underperform on Answer Refusal? A brief failure analysis or examples would help clarify the reliability trade-off.

2 comments
AuthorsOpenReview ↗

Response to Reviewer VYRm (Part I)

Dear Reviewer VYRm,

We sincerely thank you for recognizing our work, which greatly encourages us. We also appreciate your valuable suggestions regarding the quality of interaction flow recovery and the reliability of Answer Refusal. We hope our point-by-point responses below will address your concerns.

W1 & Q1: The Quality of Recovered Interaction Flow

Thank you for your valuable suggestion. We agree that evaluating the quality of recovered interaction flow is important for validating the effectiveness of IGMem. We have conducted a quality evaluation in Appendix C.2, but placed it there due to space limitations. We will give this important analysis greater prominence in the revised paper. Below, we provide a clearer explanation of the experimental setup and results.

Experimental Setup

We randomly sample 1,000 turns from H2HMem and evaluate the compiled contexts generated by GPT-4.1-nano and GLM-5.3-Flash. Since reference uptake contexts are unavailable, we conduct a model-assisted evaluation against the original conversational evidence, followed by human verification.

For each case, GPT-6 Astra receives the current turn, its compiled context, and the original textual and visual evidence underlying the selected antecedents. This also includes original sources inherited through intermediate turns. The current turn determines relevance, while factual support is checked against the original evidence rather than generated contexts alone. We further perform human verification of the evaluation results.

We evaluate three criteria to assess the information retained after source recovery:

  1. Factual grounding: Whether claims in the compiled context are supported by the original sources.
  2. Attribution and qualification: Whether speaker attribution, negation, uncertainty, conditions, and conflicting claims are preserved.
  3. Relevant-evidence coverage: Whether response-relevant source information absent from the current utterance is retained.

Experimental Results

The following table reports the pass rates for each criterion.

Construction BackboneFactual GroundingAttribution and QualificationRelevant-evidence Coverage
GPT-4.1-nano84.3%79.7%65.8%
GLM-5.3-Flash92.6%96.1%81.6%

Results and Analysis

With GLM-5.3-Flash, 92.6% of compiled contexts pass factual grounding, 96.1% preserve attribution and qualification, and 81.6% retain relevant evidence absent from the current turn. These results show that most compiled contexts preserve source-supported information, speaker attribution, and response-relevant details.

Compared with GPT-4.1-nano, GLM-5.3-Flash improves the three pass rates by 8.3, 16.4, and 15.8 percentage points, suggesting that compilation quality also depends on the construction backbone.

Relevant-evidence coverage remains lower than the other two criteria for both backbones. This may result from missed antecedents during flow recovery or omitted details during turn-conditioned compilation. Failures in grounding and attribution may also arise from compilation errors or errors inherited through recursive propagation.

Overall, these results provide direct evidence of the information quality achieved by combining Conversation Flow Recovery and Recursive Individual Uptake Compilation. They support IGMem's ability to retain relevant historical evidence while preserving factual grounding and attribution.

AuthorsOpenReview ↗

Response to Reviewer VYRm (Part II)

W2 & Q2: Answer Refusal Consistently Regresses

Thank you for your valuable suggestion. We agree that reliable answer refusal is important for persistent memory systems. A detailed analysis of Answer Refusal (AR) failures can help us better understand the limitations of IGMem. We will include this discussion in the revised paper and provide our analysis below.

Answer Refusal Analysis

Experimental Setup and Error Categories

We manually analyze AR failures on H2HMem with GLM-5.3-Flash. We classify the failures into four categories:

  1. Content hallucination: The compiled context contains information unsupported by the original evidence, leading to an incorrect answer.
  2. Detail omission: The compiled context omits important details from the original evidence, affecting the final answer.
  3. LLM hallucination: The answer model produces an incorrect answer despite no identified errors in the retrieved context.
  4. Annotation quality: Ambiguous questions make it difficult to determine whether refusal is required.

We compare three methods: Full (Text), which achieves the highest AR score among baselines; MuRAG, which achieves the highest Overall score among baselines; and IGMem.

Experimental Results

The following table reports the number of failed AR cases in each category.

MethodOverallAR FailuresContent HallucinationDetail OmissionLLM HallucinationAnnotation Quality
Full (Text)0.44872400213
MuRAG0.69005200475
IGMem0.76055822495

Results and Analysis

Among the 58 AR failures of IGMem, content hallucination and detail omission each account for only 2 cases (3.4%). Both types occur during the compilation of textual and visual evidence.

For content hallucination, we find cases where the LLM incorrectly interprets visual information when integrating it with textual evidence. For example, a user mentioned leaving a positive review for a store, while an attached image showed the store's overall rating. During compilation, the LLM incorrectly attributed this rating to the user. For detail omission, we observe cases where the LLM overlooks time-related restrictions shown in images, leading to unsupported inferences during answer generation.

In contrast, 49 failures (84.5%) are attributed to LLM hallucination during answer generation, suggesting that memory construction errors have a limited impact on AR performance. Similar patterns appear in Full (Text) (87.5%) and MuRAG (90.4%), indicating that this issue is not specific to IGMem.

Although IGMem improves the Overall score from MuRAG's 0.6900 to 0.7605, its AR performance remains lower. The main results show a similar trend: methods with higher Overall scores often achieve lower AR scores. One possible explanation is that more coherent and compact contexts encourage the answer model to infer details beyond the available evidence.

Potential Improvements

To address answer-generation hallucinations, a possible direction is to introduce an evidence-sufficiency check before answering. Key claims should be supported by traceable textual or visual evidence, with explicit refusal when such evidence is missing. For construction errors, we could further preserve speaker attribution, temporal conditions, and uncertainty during compilation. Source-based verification after compilation may also reduce misattribution and unsupported inference.

We leave these improvements toward evidence verification and reliable answer refusal for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.