- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces StreaMem, a benchmark for evaluating multimodal models’ online video understanding and long-term memory under continuous, fixed-rate inputs. It also proposes SOAM, a training-free system combining event-based memory construction, identity-aware graph memory, and question-dependent retrieval. Comparisons with existing models, ablations, and evaluation on an additional benchmark demonstrate improvements in answer accuracy and memory persistence.
Critical strengths
-
The benchmark keeps video inputs running while the model answers questions, so it can measure delays and memory retention under a more realistic workload. This is useful for understanding whether a model can keep up with a continuous stream, rather than just answer questions correctly.
-
SOAM improves accuracy and memory retention over its base model without additional training. These results show a practical benefit from adding external memory to an existing model.
Critical weaknesses
-
The experiments show that external memory helps, but do not clearly establish the benefit of SOAM’s specific design. The router ablation disables memory retrieval entirely, rather than comparing routing with always retrieving. The graph ablation also removes the identity registry, so it does not separate the benefit of identity information from that of graph structure. Given the close prior memory architectures, these results leave unclear whether the proposed design offers a meaningful advantage over a simpler memory system.
-
The evidence that memory construction keeps pace with the stream is incomplete. The paper uses the small APL–ARL gap to argue that no backlog accumulates, but this measures question-handling delays rather than the progress of asynchronous memory construction. A system could answer from an outdated memory bank while still showing little queueing delay. Without measuring memory-update lag during continuous operation, the reported results do not establish this central real-time claim.
Questions for authors
-
How do you verify that memory construction keeps pace with the incoming stream? Does APL–ARL capture delays in the memory-generation pathway, and have you measured the time between an event ending and its memory becoming available for retrieval?
-
On the existing ablation subset, how does the Router compare with always retrieving from memory, with the other settings unchanged? The current ablation disables retrieval entirely, so it does not isolate the Router’s contribution.
-
Can you compare the graph memory against a flat memory that retains the same captions and identity registry? This would help determine whether the improvement comes from the graph structure or simply from preserving identity information.