acceptodds
Withdrawn from ICLR 2027

OmniMemory: Towards Unified Omnimodal Memory for Long-Horizon Interactive Assistants

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon interactive assistants must repeatedly reason over audiovisual experiences that accumulate beyond finite context windows. Existing memory methods often prioritize visual observations and speech transcripts, underrepresenting richer auditory cues. Fragmented context, persistent interpretation errors, and perceptual information loss further limit memory reliability. We introduce OmniMemory, a training-free framework that builds evolving audiovisual memory with a unified omnimodal model. Progressive construction propagates scene and entity context across clips to build Episodic Memory, explicitly preserving visual events, speaker associations, utterance-aligned paralinguistic attributes, environmental sounds, and music. Retrospective auditing uses later evidence to revise earlier identity assignments and update affected records, while consolidation distills reusable knowledge into Core Memory. Query-aware routing reuses textual memories and selectively revisits source audiovisual clips in Native Memory when additional perceptual evidence is needed. Evaluations on M3-Bench-Robot, M3-Bench-Web, and VideoOdyssey-AV demonstrate the highest overall accuracy among evaluated systems, with substantial gains over memory baselines on Sound and Music questions. Ablations demonstrate the benefits of auditory fields, retrospective auditing, and complementary memory sources, while efficiency analysis shows a favorable trade-off among accuracy, construction overhead, and query costs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal memory, long-horizon multimodal understanding, audio-visual understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PPPLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes OmniMemory to retain the raw audio into the memory through three mechanisms. Progressive Memory Construction helps to associate the context with entity, Retrospective Consolidation & Auditing enables stored memories to be revisited and revised and Query-Aware Memory Routing makes the revisit process more efficient. Experiment on different benchmarks shows it gets a good result.

Critical strengths
  1. It retains the raw audio and processes it to make it more efficient during the retrieval.
Critical weaknesses
  1. It looks like these three mechanisms appear to overlap with the previous work. M3-Agent also builds the identity of audio, WorldMM also revisits the memory, HippoMM also retrieves the memory in different hierarchies. The author didn’t differentiate their contribution with previous work clearly.

  2. HippoMM also keeps the raw audiovisual data for retrieval.

  3. Experiment on Table 1 shows that only one training free baseline method gets improved over the base model, which implies that training free method may be sensitive to the base model. The authors only evaluates their method with a single base model, leaving it unclear whether the reported improvements generalize to other models, such as Gemini 3.1 Pro.

  4. Progressive Memory Construction helps to build the connection between identity and information. However, OmniMemory didn’t outperform Video-RAG on personal understanding.

Questions for authors

What gap in prior work does this work address?

Reviewer_DxPNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework to capture and retrospectively audit omnimodal information, which is especially useful for sound and music questions. It divides memory into episodic memory, core memory and native memory. These memories can be updated, revised, and audited. Experiments show great improvements for non-speech scenarios.

Critical strengths
  • This paper is easy to follow and well-organized.
  • It addresses an important problem of providing auditable memory without information loss and hallucinations.
  • Extensive experiments demonstrate the strengths and limitations of the proposed framework.
Critical weaknesses
  • The main novelty of the proposed method is unclear. It is straightforward to restore the original information for auditing. The design of hierarchical memory is also widely explored.
  • Besides restoring the raw information, this paper lacks insights to achieve similar performance with less memory, time and token costs, especially for evolving lifelong agents.
  • There are works which can capture non-speech information. However, this paper only compares with those that only rely on text summaries, which is unfair.
Questions for authors

Please see the weaknesses.

Reviewer_DQLwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OmniMemory is a training-free framework for long-horizon audiovisual memory that integrates episodic, core, and native memory. It explicitly preserves auditory details and their associations with individuals, and supports subsequent question answering through cross-segment context propagation, retrospective auditing, and on-demand replay. The paper reports strong results on M3-Bench-Robot, M3-Bench-Web, and VideoOdyssey-AV, along with component ablations and efficiency analyses.

Critical strengths

The experimental results are reasonably convincing and demonstrate the system’s practical value. The method achieves strong overall performance across three benchmarks, while ablations on auditory fields, auditing, and memory sources support the effectiveness of its design choices. Although the architectural novelty is relatively limited, this integrated approach to remembering audiovisual experiences remains practically useful.

Critical weaknesses

W1. Limited architectural novelty, with the independent contribution and comparative advantage of person attribution remaining unclear. Hierarchical memory and entity association have precedents in prior work. M3-Agent also performs person attribution through face–voice equivalence relationships and includes a “w/o equivalence” ablation. While this paper emphasizes the importance of speaker attribution and cross-segment entity consistency, the “w/o audit” ablation in Table 3 only evaluates post hoc revision; it does not separately isolate the contributions of initial person attribution and cross-segment identity association. Likewise, the gains in overall performance and PU in Table 1 do not directly establish better attribution quality: PU encompasses broader person-related reasoning, and the two systems differ in both their underlying models and memory pipelines.

W2. Insufficient validation of memory for interpersonal interactions across multiple days. The current benchmarks include interactions and long videos, but evaluation is still primarily conducted on individual recordings spanning minutes to hours. This is insufficient to establish the system’s ability to maintain person identity associations and remember interactions across continuous, multi-day experiences.

Questions for authors

Q1. Could the authors separately evaluate the contributions of speaker-to-person association and cross-segment identity association while keeping the downstream memory, retrieval, and answering pipeline fixed, and compare these components with M3-Agent’s person-attribution approach on the same inputs? It would be useful to report both attribution quality and performance on identity-dependent questions, distinguishing the gains from initial attribution from those introduced by subsequent auditing, rather than relying solely on system-level PU scores.

Q2. Could the authors add an evaluation on EgoLifeQA, focusing on RelationMap, audio-required questions, and questions requiring look-back periods longer than 24 hours, or a representative subset of these categories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.