- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies agent memory for multimodal multi-party conversations, where information about one event is spread across speakers, turns and images, and different participants pick up different details of the same content. IGMem links each turn to earlier turns using recency, same-speaker and global semantic cues, then has an LLM compile a short, turn-conditioned context from those sources, which is propagated recursively and stored with the original turn. On H2HMem and Mem-Gallery with two backbones, it improves the overall LLM-Judge score by 5.5 to 10.5 points over the strongest baseline.
Critical strengths
The problem is clearly motivated, and the method follows directly from it, with each component (flow recovery, conditioned uptake, recursive propagation) shown to matter in the ablations. The gains are consistent across two benchmarks and two backbones, results are averaged over three runs, and code is provided. The uptake-quality audit in Appendix C.2 is a useful addition that many memory papers lack.
Critical weaknesses
-
Construction cost is not reported. IGMem calls a multimodal LLM once per turn (over 7,000 turns on H2HMem), while retrieval baselines such as MuRAG and NaiveRAG do no write-time processing. Figure 4 compares only query-time tokens, so the efficiency claim and the comparison with these baselines are incomplete.
-
Answer Refusal drops in every setting (by 9.4 and 12.6 points on H2HMem, 4.3 and 6.0 on Mem-Gallery). Together with the 84.3% factual-grounding rate for GPT-4.1-nano in Appendix C.2, this suggests that compiled contexts introduce unsupported information that leads the model to answer when it should not, and recursive propagation may spread these errors.
-
The core idea, enriching each memory entry with LLM-written context at write time, is close to A-Mem and contextual retrieval approaches. Appendix C.1 compares against a variant of IGMem's own pipeline rather than an existing enrichment method.
Questions for authors
My positive score depends on the following being addressed in the rebuttal:
-
Please report memory-construction cost (LLM calls, tokens and time) for IGMem and all baselines, and add a baseline that applies comparable write-time LLM enrichment to MuRAG, so the gain from interaction grounding can be separated from the gain from extra LLM processing.
-
Please analyze the drop in Answer Refusal. How often do refusal failures trace back to unsupported content in compiled contexts, and do grounding errors accumulate with recursion depth?
If these show that the gains hold under matched construction effort and that error propagation is limited, I will keep my score. Otherwise I would lower it.
5 comments
Response to Reviewer Lnd4 (Part II)
W2 & Q2: Analyze the drop in Answer Refusal. How often do refusal failures trace back to unsupported content in compiled contexts, and do grounding errors accumulate with recursion depth?
Thank you for your valuable suggestion. We agree that further analysis of the Answer Refusal (AR) performance drop would help clarify the limitations of IGMem.
In Appendix C.2, we briefly evaluated the quality of compiled contexts. We will expand this section with more detailed analysis and discussion in the revised paper.
In the following, we first analyze AR failures and then examine the sources and propagation of errors in compiled contexts.
Answer Refusal Analysis
Error categorization. To better understand the failures on Answer Refusal (AR), we classify the failure cases into four categories:
- Content hallucination: The compiled context introduces unsupported information that is not present in the original evidence, leading to an incorrect answer.
- Detail omission: The compiled context omits important details from the original evidence, affecting the final answer.
- LLM hallucination: The answer model generates an incorrect answer despite receiving a context without identified errors.
- Annotation ambiguity: The question is ambiguous, making it difficult to determine whether the model should refuse to answer.
Error distribution. We analyze the AR failure cases on H2HMem with GLM-5.3-Flash. We compare IGMem with two representative baselines: Full (Text), which achieves the highest AR score among baselines, and MuRAG, which achieves the highest Overall score among baselines. The distribution of failure types is shown below.
| Method | Overall | AR Failures | Content Hallucination | Detail Omission | LLM Hallucination | Annotation Ambiguity |
|---|---|---|---|---|---|---|
| Full (Text) | 0.4487 | 24 | 0 | 0 | 21 | 3 |
| MuRAG | 0.6900 | 52 | 0 | 0 | 47 | 5 |
| IGMem | 0.7605 | 58 | 2 | 2 | 49 | 5 |
Results and discussion. Among the 58 AR failures of IGMem, content hallucination and detail omission each account for only 2 cases (3.4%). Both types occur during the compilation of textual and visual evidence.
For content hallucination, we find cases where the LLM incorrectly interprets visual information when integrating it with textual evidence. For example, a user mentioned leaving a positive review for a store, while an attached image showed the store's overall rating. During compilation, the LLM incorrectly attributed this rating to the user. For detail omission, we observe cases where the LLM overlooks time-related restrictions shown in images, leading to unsupported inferences during answer generation.
In contrast, 49 failures (84.5%) are attributed to LLM hallucination during answer generation, suggesting that memory construction errors have a limited impact on AR performance. Similar patterns appear in Full (Text) (87.5%) and MuRAG (90.4%), indicating that this issue is not specific to IGMem.
Although IGMem improves the Overall score from MuRAG's 0.6900 to 0.7605, its AR performance remains lower. The main results show a similar trend: methods with higher Overall scores often achieve lower AR scores. One possible explanation is that more coherent and compact contexts encourage the answer model to infer details beyond the available evidence.
Error Accumulation in IGMem
Compiled-context quality. In Appendix C.2, we evaluate 1,000 randomly sampled turns from H2HMem using GPT-4.1-nano and GLM-5.3-Flash. GPT-6 Astra assesses each compiled context against the original textual and visual evidence, including sources inherited through intermediate turns, followed by human verification. We examine three criteria: (1) factual grounding, whether claims are supported by original evidence; (2) attribution and qualification, whether speaker attribution and factual qualifications are preserved; and (3) relevant-evidence coverage, whether response-relevant information is retained. With GLM-5.3-Flash, the pass rates are 92.6%, 96.1%, and 81.6%, respectively, indicating generally reliable compilation but remaining errors and omissions.
Error attribution and propagation. We further examine the 74 cases that fail factual grounding with GLM-5.3-Flash. Among them, 69 involve direct compilation errors, where the LLM introduces unsupported information from original evidence, while only 5 involve inherited errors from antecedent contexts. We also trace error propagation to subsequent turns. These 74 turns serve as antecedents for 192 later turns, but only 15 (7.8%) inherit the incorrect information.
Results and discussion. These results suggest that most grounding errors arise from direct compilation rather than recursive error accumulation. Error propagation is also limited, as turn-conditioned uptake compilation selectively retains response-relevant information, potentially filtering out incorrect details inherited from antecedents.
Response to Reviewer Lnd4 (Part III)
W3: Comparison between our core idea and A-Mem and other contextual retrieval approaches
Thank you for your question. In Section 4.2, we compare IGMem with A-Mem and other contextual memory methods. However, the key design differences may not have been sufficiently explained due to space limitations. We will clarify these differences in the revised paper.
Comparison with Existing Methods
Enriching memory entries with LLM-generated context is a common goal of memory systems, but existing methods differ in how they achieve it.
A-Mem. A-Mem uses LLMs to generate keywords, tags, and contextual descriptions for memory notes. It retrieves semantically related notes and uses LLMs to link and update them. However, semantic similarity may miss interleaved dependencies, while note linking does not explicitly capture individual uptake. IGMem instead combines multiple cues to recover antecedents and selects source details conditioned on the current response.
Mem0. Mem0 extracts important facts from recent messages and conversation summaries. It retrieves related memories and uses LLMs to add, update, delete, or retain facts. However, this process may miss distant dependencies and lose the connection between responses and their specific sources. IGMem instead recovers cross-turn antecedents and preserves response-specific evidence through recursive context propagation.
MemGPT. MemGPT manages information through hierarchical context and external storage. It recursively summarizes older messages and retrieves historical records when needed. However, hierarchical storage and summarization do not explicitly address interleaved dependencies or individual uptake. IGMem instead organizes distributed evidence around each response and propagates relevant context across turns.
MIRIX. MIRIX organizes multimodal information into six memory types using specialized managers. A Meta Memory Manager coordinates memory storage and retrieval across these types. However, this organization does not explicitly capture cross-participant dependencies or how different responses use the same source. IGMem instead recovers response-relevant antecedents and selectively compiles the textual and visual details taken up by each response.
IGMem. The key difference lies in how contextual information is selected and propagated. IGMem combines three mechanisms: multi-cue antecedent selection, turn-conditioned uptake compilation, and recursive context propagation. Together, they organize distributed interaction evidence and response-specific details into retrievable memory units.
Overall Performance Comparison
Tables 1 and 2 show that IGMem consistently outperforms A-Mem and other memory baselines in Overall scores across both benchmarks and backbones.
On H2HMem, IGMem improves Overall scores over A-Mem from 0.5757 to 0.6841 (+10.84 percentage points) with GPT-4.1-nano, and from 0.5965 to 0.7605 (+16.40 points) with GLM-5.3-Flash. On Mem-Gallery, the corresponding improvements are from 0.7344 to 0.8042 (+6.98 points) and from 0.8214 to 0.8939 (+7.25 points).
These results support the benefit of interaction-grounded memory construction beyond existing contextual enrichment methods. The ablation studies in Section 4.3 further demonstrate the contributions of each component.
Response to Reviewer Lnd4 (Part I)
Dear Reviewer Lnd4,
We sincerely thank you for recognizing our work and for your generous rating, which greatly encourages us. We also appreciate your questions and suggestions and hope our point-by-point responses address your concerns.
W1 & Q1: Memory-construction costs and comparable write-time enrichment for MuRAG
Thank you for raising this important issue. We reported construction costs for LLM-based methods in Appendix C.4 due to space limits. We will move this analysis into the main text. Below, we compare construction costs across methods and test MuRAG with a comparable write-time LLM budget.
Memory-construction Cost Analysis
Experimental setup and metrics. We use H2HMem with GLM-5.3-Flash and the construction settings from our main experiments. All methods build memory from scratch on the same 25 conversation histories. Experiments run on a server with an NVIDIA H20 GPU and a 64-core CPU (128GB RAM). LLM calls use OpenRouter; the local GPU handles embedding encoding and similarity computation.
For each history, we sum construction-time LLM calls and API-reported
prompt_tokensandcompletion_tokens, then average across histories. Image captions for text-based methods are provided by H2HMem, so their generation costs are excluded, although they would incur costs in practice. We report LLM calls and tokens instead of wall-clock time, which depends on API load, network conditions, etc.Experimental results. Values are averaged per history; M denotes million tokens.
Result analysis. IGMem uses 333.24 calls and 1.036M total tokens, 17.7% fewer than Mem0 and 62.1%–85.5% fewer than A-Mem, MemGPT, and MIRIX. Its input cost (0.955M) is close to A-Mem (0.975M) and above Mem0 (0.854M). The main saving comes from output tokens (0.081M, 80.0%–95.4% fewer), as IGMem preserves original turns and generates only compact, response-relevant contexts.
NaiveRAG, MuRAG, and NGM need no construction-time generative LLM calls under our settings. IGMem adds a one-time construction cost to organize distributed evidence, but achieves better QA performance at comparable query-time token budgets (Section 4.4, Figure 4). This follows the common design of agent memory systems, which invest in memory construction to improve subsequent query efficiency and accuracy.
Comparison with LLM-enriched MuRAG
LLM-enriched MuRAG. We extend MuRAG with local-window context enrichment. For each turn, we provide an LLM with the current message and information of its preceding eight turns. The LLM generates an enriched message while preserving source-supported facts, speaker attribution, temporal conditions, and uncertainty. Enriched text is limited to twice the original length. The window size is chosen to maintain a construction cost comparable to IGMem. The enriched text replaces the original text and is re-encoded for retrieval. Query processing follows the original MuRAG pipeline, retrieving enriched records by similarity and supplying them to the answer model.
Experimental setup and results. We use H2HMem with GLM-5.3-Flash and the main experimental settings. Results are averaged over three runs.
Result analysis. LLM-enriched MuRAG uses 1.040M construction tokens versus IGMem's 1.036M (0.4% difference). With Top-6 retrieval, enrichment lowers MuRAG's Overall from 0.7034 to 0.6811 (−2.23 pp) and raises query tokens from 2,167 to 2,649 (+22.2%). IGMem reaches 0.7605 (7.94 pp higher) using 2,095 query tokens (20.9% fewer).
This suggests that additional LLM processing and longer contexts do not necessarily improve QA performance. Local-window enrichment may introduce irrelevant information from interleaved conversations or omit important details when rewriting original records. IGMem instead recovers antecedents through temporal, participant, and semantic cues, and compiles response-relevant context while preserving original observations. This allows relevant evidence to remain jointly accessible for downstream retrieval and reasoning.