acceptodds
Withdrawn from ICLR 2027

StreaMem: Can MLLMs Understand Real-World Streams with Long-Term Memory?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deployment of multimodal large language models (MLLMs) demands systems that process continuous multimodal streaming inputs at a fixed rate, respond instantly, and retain dynamically evolving long-term memory, capabilities that current MLLMs still largely lack. To rigorously assess such capabilities, we build **StreaMem**, the first benchmark to jointly evaluate MLLMs' real-time processing and long-term memory under a real-streaming protocol, covering four scenarios and 12 subtasks with 210 videos of more than 30 minutes and over 3,000 open-ended QA pairs, together with four metrics. Evaluation of 7 existing models on StreaMem confirms that real-time processing and long-term memory remain unsolved under real-streaming conditions. We then propose **SOAM**, a **S**treaming **O**mni **A**gentic **M**emory that equips MLLMs with streaming understanding and external long-term memory without training. SOAM parses the incoming stream into semantically coherent events via event boundary detection, organizes them into a heterogeneous two-layer memory graph with biometric entity anchoring, routes each question through an LLM Router that decides whether memory retrieval is needed, and retrieves evidence along complementary semantic, entity co-occurrence, and temporal-anchor paths. SOAM achieves state-of-the-art performance among open-source models and methods of comparable scale, with substantial improvements in long-term memory persistence. The code is anonymously available at https://anonymous.4open.science/r/SOAM-FB1324wrc/, and the dataset will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Streaming Memory, MLLMs, Video Understanding, Benchmark Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wrpbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StreaMem, a benchmark for evaluating multimodal models’ online video understanding and long-term memory under continuous, fixed-rate inputs. It also proposes SOAM, a training-free system combining event-based memory construction, identity-aware graph memory, and question-dependent retrieval. Comparisons with existing models, ablations, and evaluation on an additional benchmark demonstrate improvements in answer accuracy and memory persistence.

Critical strengths
  1. The benchmark keeps video inputs running while the model answers questions, so it can measure delays and memory retention under a more realistic workload. This is useful for understanding whether a model can keep up with a continuous stream, rather than just answer questions correctly.

  2. SOAM improves accuracy and memory retention over its base model without additional training. These results show a practical benefit from adding external memory to an existing model.

Critical weaknesses
  1. The experiments show that external memory helps, but do not clearly establish the benefit of SOAM’s specific design. The router ablation disables memory retrieval entirely, rather than comparing routing with always retrieving. The graph ablation also removes the identity registry, so it does not separate the benefit of identity information from that of graph structure. Given the close prior memory architectures, these results leave unclear whether the proposed design offers a meaningful advantage over a simpler memory system.

  2. The evidence that memory construction keeps pace with the stream is incomplete. The paper uses the small APL–ARL gap to argue that no backlog accumulates, but this measures question-handling delays rather than the progress of asynchronous memory construction. A system could answer from an outdated memory bank while still showing little queueing delay. Without measuring memory-update lag during continuous operation, the reported results do not establish this central real-time claim.

Questions for authors
  1. How do you verify that memory construction keeps pace with the incoming stream? Does APL–ARL capture delays in the memory-generation pathway, and have you measured the time between an event ending and its memory becoming available for retrieval?

  2. On the existing ablation subset, how does the Router compare with always retrieving from memory, with the other settings unchanged? The current ablation disables retrieval entirely, so it does not isolate the Router’s contribution.

  3. Can you compare the graph memory against a flat memory that retains the same captions and identity registry? This would help determine whether the improvement comes from the graph structure or simply from preserving identity information.

Reviewer_7J7oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a benchmark, StreaMem, to evaluate MLLMs' real-time processing and long-term memory capabilities. The authors claim this is the first benchmark in this specific setting and propose a systematic pipeline to construct and annotate the dataset. The benchmark covers four domains: episodic perception, content understanding, real-time reasoning, and memory evolution. They also propose SOAM, a training-free framework that builds external memory from video events and retrieves relevant information for question answering. The results show SOAM achieves SOTA performance.

Critical strengths

The paper addresses an important practical problem: understanding continuous video streams while maintaining long-term memory and real-time response, which is valuable for real-world understanding and embodied intelligence.

Critical weaknesses
  1. The experiments do not clearly establish SOAM's overall advantage in practical streaming settings. While SOAM uses more hardware resources, it only improves FMA by only 1.47 points while increasing average processing latency from 1.74 to 19.25 seconds, compared with MiniCPM-o.
  2. Given the difference in hardware resources, the claim of "state-of-the-art performance among ... of comparable scale" is not sufficiently supported. And the response time of SOAM (19.25 seconds) is far away from the claim "real-time".
Questions for authors
  1. Can you provide a real-time demo to demonstrate SOAM’s main advantages over existing methods?

  2. Figure 3 contains many details and icons, making the overall workflow difficult to follow. Can you simplify the diagram to make SOAM’s structure and information flow clearer? And the Section 3.1 contains too many concepts and is hard to read, can you use a diagram in the paper rather than in the appendix?

  3. Can you explain why biometric features are used for entity association, how SOAM handles videos without recognizable faces, and whether more general visual features could be used instead?

Reviewer_t3vWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StreaMem, a benchmark for evaluating real-time streaming video understanding and long-term memory in MLLMs. It comprises 210 videos and over 3,000 QA pairs across 12 subtasks, along with a real-streaming evaluation protocol that accounts for asynchronous processing and response latency. The authors also propose SOAM, a training-free memory framework that combines event-based segmentation, entity-centric memory graphs, and multi-path retrieval. Experiments demonstrate improvements in answer accuracy and memory persistence over existing open-source methods.

Critical strengths
  1. Important and Practical Problem: The paper addresses the important challenge of enabling MLLMs to continuously process streaming multimodal inputs while maintaining long-term memory and responding to real-time queries. This is a practically relevant problem for real-world AI assistants and embodied agents.

  2. Valuable Benchmark Contribution: StreaMem provides a valuable benchmark with diverse video scenarios and 12 subtasks covering perception, understanding, reasoning, and memory evolution. The dataset offers a useful foundation for systematically evaluating streaming understanding and long-term memory capabilities.

  3. Realistic Streaming Evaluation Protocol: A notable strength is the real-streaming evaluation protocol, which decouples data delivery from model processing. Unlike conventional pseudo-streaming evaluation, this design captures asynchronous processing, queue accumulation, and response latency, providing a more realistic assessment of streaming systems.

Critical weaknesses
  1. Lack of Comparison with Strong and Simple Baselines: Recent work, SimpleStream [1], demonstrates that a straightforward approach using only a few recent video frames, without explicit memory construction or retrieval, can outperform more sophisticated streaming video understanding methods. Notably, when considering the nine shared tasks on OVO-Bench, SimpleStream achieves an average score of 67.70%, compared to 62.82% for SOAM. Although these results are obtained using different backbones and inference configurations, they highlight the need for a controlled comparison to better establish the effectiveness and necessity of the proposed memory architecture. Therefore, a direct comparison with SimpleStream under the same backbone and evaluation settings would be valuable. More importantly, SimpleStream should also be included as a baseline on the proposed StreaMem benchmark to quantify the benefits of explicit long-term memory over a simple recent-frame approach.

  2. Insufficient Evaluation on Existing Streaming Video Understanding Benchmarks: Although SOAM is proposed as a framework for streaming video understanding, its evaluation on established streaming benchmarks is limited to OVO-Bench. Additional evaluations on other widely used benchmarks, such as StreamingBench [2], would strengthen the empirical validation of the proposed method. In particular, apples-to-apples comparisons with SimpleStream and other competitive methods under consistent backbone and evaluation settings would help establish SOAM's generalizability and provide a clearer understanding of the accuracy–efficiency trade-offs introduced by its additional memory construction and retrieval mechanisms.

  3. Limited Evaluation of Long-Term Streaming Capabilities: The videos in StreaMem have an average duration of approximately 50 minutes, with 86% falling within the 30–60 minute range. While these videos are longer than those in many conventional video understanding benchmarks, their durations remain relatively limited compared to real-world long-term streaming scenarios, such as EgoLife [3], which involves continuous experiences spanning hours or days. Since SOAM continuously accumulates memories as the stream progresses, evaluating its performance on substantially longer streams would be important. In particular, further analysis of memory scalability, retrieval accuracy, and processing efficiency over extended durations would provide stronger evidence for the applicability and scalability of SOAM in real-world long-term streaming settings.

  4. Insufficient Clarity in Method Description and Visualization: SOAM consists of several interconnected components, but their detailed operations and interactions are not always clearly presented. In particular, Figure 3 plays a central role in illustrating the proposed framework, yet its cluttered layout, unclear data flow, and limited visual clarity make the overall architecture difficult to follow. Providing more explicit pseudocode and a clearer, more carefully structured architectural diagram would substantially improve the clarity and reproducibility of the proposed method.

  5. Limited Analysis of Dataset Annotation Quality: The QA pairs in StreaMem are initially generated by an MLLM and subsequently verified by human annotators. While the authors describe the human verification process, the paper provides limited quantitative evidence of annotation quality and consistency, such as inter-annotator agreement or annotation correction and rejection rates. Given the importance of reliable ground-truth annotations, particularly for temporally complex reasoning and memory evolution tasks, a more systematic analysis of annotation quality would strengthen the credibility of the proposed benchmark.

References

[1] Shen, Yujiao, et al. "A Simple Baseline for Streaming Video Understanding." arXiv preprint arXiv:2604.02317 (2026).

[2] Lin, Junming, et al. "StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding." ICASSP, 2026.

[3] Yang, Jingkang, et al. "EgoLife: Towards Egocentric Life Assistant." CVPR, 2025.

Questions for authors

Could the authors provide a controlled comparison between SOAM and SimpleStream using the same backbone, particularly on StreaMem and the shared OVO-Bench tasks? Additionally, would it be possible to evaluate SOAM on StreamingBench to further demonstrate its generalizability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.