acceptodds
Withdrawn from ICLR 2027

AMBER: Anytime Multimodal Memory Bridging Episodes via Rich Association

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal conversational memory systems struggle with required evidence when information is updated, extended, or contradicted across episodes, where previous similarity-based retrieval failed. To address this challenge, we introduce AMBER (Anytime Multimodal Memory Bridging Episodes via Rich Association), which builds graph with temporal, procedural, complementary, and conflicting associations, continually organizing episodes into paths centered on shared association anchors. When association retrieval is needed, AMBER adaptively activates one type for the query, traverses matching paths from initial semantic matches, and ranks the resulting evidence chains under a fixed budget. To evaluate whether memory systems can preserve required evidence while incorporating new information, we introduce MemStream, annotated 1,603 information states for 779 evolution-related questions with reference answers and supporting evidence from Mem-Gallery and H2HMem. MemStream enables streaming evaluation with process-level metrics for answer continuity and degradation, revealing an average 58.48% decrement in state F1 compared to the final. Across 2 widely used MM benchmarks and our proposed one, AMBER improves average F1 by 2.68% over the strongest of four baselines, while recovering 19.07% more evidence in cross-episode queries.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Conversational Memory, Memory Evolution, Evidence Retrieval, Long-Term Memory, Streaming Evaluation, State-Aware Query Processing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F2nQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AMBER, a multimodal conversational memory framework that builds temporal, procedural, complementary, and conflicting associations across episodes. At retrieval time, it selects an association type, expands evidence chains from semantic seeds, and ranks them under a fixed budget. The paper also introduces MemStream, a streaming evaluation derived from Mem-Gallery and H2HMem with state-level metrics for continuity and degradation. AMBER improves average F1 over the strongest baseline by 2.68% and improves evidence recall on evolution-related queries.

Critical strengths

The motivation is reasonable: independent similarity retrieval can miss evidence that is procedurally, temporally, or contradictorily related to the current query. The proposed association taxonomy is intuitive, and MemStream is a useful attempt to evaluate process-level memory behavior rather than only final answers. The paper evaluates two answer backbones and reports both answer and evidence-retrieval metrics.

Critical weaknesses

The method is fairly complex while the demonstrated gains are modest. The average F1 improvement over the strongest baseline is 2.68%, and key ablations provide limited support for the necessity of the proposed machinery: removing specific association reduces F1 by only 0.47%, while removing anchor generation reduces it by only 0.27%. The method also relies heavily on LLM-assisted association construction and anchor generation without sufficient analysis of cost, latency, reproducibility, or failures with weaker/open models. MemStream's annotation agreement and metric robustness are also under-validated, and the small judge-based gains need statistical significance analysis.

Questions for authors
  1. What is the total memory-construction and retrieval cost of AMBER, including all LLM calls?
  2. How does AMBER perform if association construction and anchor generation use an open or weaker model?
  3. Are the answer-quality gains statistically significant under paired tests?
  4. What is the annotation agreement for MemStream state labels and supporting evidence?
1 comment
Reviewer_F2nQOpenReview ↗

Post-rebuttal follow-up

I have not seen a substantive response or additional experiment addressing the questions in my review. My concerns therefore remain unchanged. In particular, the framework introduces considerable complexity while showing limited average improvement over the strongest baselines, and the reported ablations for association routing and anchor generation have relatively small effects. The paper also lacks sufficient analysis of total construction cost, retrieval latency, association construction with weaker or open models, statistical significance, and MemStream annotation consistency. I maintain my rating of 1 (Clear Rejection).

Reviewer_ppsCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an AMBER method that organizes multimodal dialogue segments into paths with temporal, procedural, complementary, or conflicting relations. At query time, it selects a relation mode and an anchor, expands paths from a semantic seed, and ranks evidence chains under a budget of ten dialogue segments. Additionally, the paper presents MemStream, which leverages state-specific questions and supporting evidence to evaluate questions over successive dialogue prefixes. Experiments compare four memory baselines on two source benchmarks and evaluate MemStream using two answer backbones.

Critical strengths

S1. MemStream seems intuitive: a system might correctly answer when the full history is available, but fail early in the same trajectory. State-specific references and separate evidence recall metrics make it possible to examine these failures, rather than relying solely on final answer scores.

S2. The evidence gain associated with evolution is meaningful. The paper separates evidence coverage from answer correctness and reports unsuccessful robustness findings, rather than treating every test as a success.

Critical weaknesses

W1. The contributions of MemStream require more robust validation of its annotation and evaluation cohorts. The paper reports no consistency or reliability metrics. Annotation quality and cohort sensitivity are crucial criteria for determining whether this benchmark supports evolving memory evaluation; however, as it stands, it appears insufficient.

W2. According to the paper, evidence for isolating specific routing associations is scarce and difficult to trace. In the H2HMem comparison, removing associative selection or anchor generation only reduced the F1 score by 0.47 and 0.27, respectively; the full method outperformed similarity-based retrieval by 0.76. Additionally, I examined the supplementary material, where different ablation queues were marked as partially completed, and incomplete results were excluded, raising potential reproducibility concerns.

Questions for authors

Q1. Has the data used been effectively annotated and independently verified? What consistency or audit results are available for reference?

Q2. Which published result files and configurations were used to generate each row in Table 2a? How are they related to the partial ablations?

Reviewer_LPWfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AMBER, a multimodal memory framework that organizes arriving episodes into paths by explicit cross-episode association type and retrieves evidence as ranked chains under a fixed budget. It also introduces MemStream, a streaming benchmark with state-specific answers and process-level metrics for how evidence access and answer quality evolve as information changes.

Critical strengths
  1. The problem framing is the right one for a real assistant, because information in a long conversation is updated, extended, and contradicted, and a memory system that only retrieves the most similar current episodes will lose the earlier evidence that is still needed.
  2. MemStream is a strong contribution in the paper, because streaming prefixes, state-specific answers, and process metrics for continuity, retention, degradation, and hidden earlier failure diagnose failures that a final answer score cannot separate.
  3. The method is coherent as a whole, with online path construction, one query-selected association type, anchor matching, chain-level ranking, and assembly that preserves chain order and association context.
  4. It beats the strongest included baseline in every main table combination, and the improvement is larger where it should be, on evolution-related queries, where both current support recall and retained earlier evidence recall rise substantially.
  5. The appendix is transparent and helpful, because it reports the single-type limitation, the scenario variation, the counterfactual null result, and the added construction cost instead of hiding them.
Critical weaknesses
  1. Novelty is incremental. Typed multi-relation graphs and query-conditioned routing are already established, with MAGMA using temporal, causal, semantic, and entity graphs with an intent-aware router, MRAgent using query-conditioned expansion and pruning, and TRACE using typed temporal, causal, update, and contradiction relations with validity-aware paths. What is new here is the specific combination of four association types, anchor-based chain ranking, and streaming process evaluation in the multimodal conversational setting, not a new paradigm.
  2. Its state of the art claim is incomplete, because the main baselines are MuRAG, Neural Graph Memory, AUGUSTUS, and UniversalRAG, while the closest current systems it cites, including HippoRAG 2, MRAgent, MAGMA, and V-Mem, are not run as baselines. I therefore cannot tell from the paper whether AMBER still wins against current state of the art.
  3. The core routing ablations are small. On the H2HMem rerun, removing a specific association type costs only 0.47 F1 points and removing anchor generation costs only 0.27 points, and overall evidence retrieval barely moves, so the headline gains look more concentrated in the evolution subset and in chain assembly than in routing itself.
Questions for authors
  1. Why are the routing ablations so small on H2HMem while the MemStream evolution gains are large, and is the main benefit coming from chain ranking and assembly rather than from selecting one association type?
  2. How robust is anchor matching to paraphrase, noisy anchors, and controller or activator errors, and what are the controller and activator accuracies on their own decisions?
  3. What does anytime mean in this paper, and if it does not mean interruptible answering under a time budget, would you clarify that in the title or framing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.