acceptodds
Withdrawn from ICLR 2027

EventMem: Adaptive Event-Centric Memory for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

MLLMs often struggle with real-time reasoning over streaming videos as visual information continuously accumulates over time. A key bottleneck is efficient memory management: preserving long-term context within a compact representation while retaining transient and fine-grained visual evidence. To address these challenges, we propose EventMem, a training-free event-centric memory framework: (1) EventMem discovers structured event units from incoming video streams by temporal-aware adaptive K-means event segmentation (TAKC). (2) it performs temporal segmentaion and aligned frame compression (TSAFM) to each event, reducing redundant visual information while retaining frames. (3) it builds complementary event-residual memory to preserve both long-term context and recent fine-grained evidence, and retrieves relevant historical events and fuses them with short-term evidence for efficient streaming reasoning. Extensive experiments show that EventMem consistently improves streaming video understanding. It improves object search, long-term memory, and short-term memory by 11.71, 6.73, and 3.17 points, respectively, validating the effectiveness of event-aware compression and complementary memory for streaming reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Event-Centric Memory, Visual Token Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_orbrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EventMem, a training-free event-centric memory framework for streaming video understanding with frozen MLLMs. The method consists of three main components. First, Temporal-aware Adaptive K-Means Event Segmentation (TAKC) clusters frame-level features and adaptively determines the number of events. Second, Temporal Segmentation and Aligned Frame Compression (TSAFM) preserves frames with large temporal changes while merging tokens from relatively stable frames. Third, the method maintains a complementary long-term event memory and short-term residual memory, retrieving query-relevant historical events while retaining recent fine-grained visual evidence. The method is evaluated on StreamingBench Real-Time, StreamBench, and OVO-Bench Real-Time using LongVA and Qwen2.5-VL backbones. The reported results show consistent improvements over the corresponding backbones, with particularly noticeable gains on object search and long-term memory tasks. The ablation study also suggests that adaptive event segmentation, token compression, and residual short-term memory contribute complementary improvements.

Critical strengths
  1. The long-term event memory plus short-term residual memory design is simple, intuitive, and supported by the ablation study.

  2. EventMem is training-free and produces consistent improvements across several streaming-video benchmarks and backbones.

Critical weaknesses
  1. The event segmentation mechanism is not sufficiently justified. TAKC determines the number of events from K-means clustering of frame-level visual features, but the number of feature-space clusters does not necessarily correspond to the number of temporal events: a single event may contain multiple visual states, while distinct events may have highly similar frame appearances. This concern is further amplified by the fixed K_{\max}. I am also unclear about the boundary criterion in Eq. (5): maximizing the summed distance to two representative frames does not intuitively correspond to locating the transition boundary. Since event discovery is a central contribution, this needs stronger justification or clarification.

  2. The efficiency claim is insufficiently evaluated. The paper motivates EventMem as an efficient streaming-memory framework, but reports almost exclusively task accuracy. It does not provide token counts, compression ratio, memory usage, latency, throughput, or comparisons under matched memory/token budgets. Since EventMem also stores event tokens, key-frame tokens, captions, and residual raw-frame tokens, it is difficult to tell whether the gains come from better memory organization or simply from retaining more information.

Questions for authors
  1. Could the authors clarify Eq. (5)? Why is the event boundary defined by maximizing the summed distance to two representative frames, and is this exactly what is used in the implementation?
  2. What justifies treating the silhouette-selected number of feature clusters as the number of temporal events? A single event may contain multiple visual states, while distinct events may have similar frame-level features. It would be useful to know how well the selected K^* correlates with actual event boundaries or event counts.
  3. Can the authors provide efficiency results under matched memory/token budgets? Retained token counts, compression ratio, memory usage, and latency would help verify the claimed streaming efficiency. The authors should also reconcile the numerical inconsistencies between several tables and the corresponding text.
Reviewer_YMMLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EventMem, a training-free framework for streaming video understanding. It combines adaptive event segmentation, change-aware visual token compression, and complementary long-term and short-term memory. Historical events are retrieved and combined with recent visual information for question answering. Experiments on three benchmarks show competitive results against existing training-free methods.

Critical strengths
  1. The paper addresses an important problem in streaming video understanding. The proposed framework is reasonably designed and does not require additional training.

  2. The experiments cover three benchmarks and a good range of recent baselines. EventMem achieves competitive results among training-free methods, and the ablation studies support the effectiveness of its main components.

Critical weaknesses
  1. Limited novelty. The proposed components mainly build on existing techniques, including K-means clustering, similarity-based token merging, and long-term/short-term memory. While their combination is reasonable, the technical contributions over recent methods such as OASIS and FreshMem are relatively incremental.

  2. Lack of efficiency analysis. Efficient memory management is a central motivation of the paper, but important measurements such as token compression ratio, GPU memory consumption, and inference latency are missing. Although the method achieves competitive accuracy, some improvements over existing methods are marginal. It is unclear whether the additional complexity is justified, especially under the same memory or token budget.

Questions for authors
  1. Can the authors provide efficiency comparisons, including compression ratio, memory usage, and inference latency, preferably under the same token budget?

  2. How does TAKC compare with simpler temporal segmentation methods? Since K-means ignores temporal ordering, how does it handle repeated events?

  3. How does the proposed weighted token merging compare with simple averaging or similarity-based frame selection? Is the additional complexity necessary?

Reviewer_1vMKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EventMem, a training-free memory framework for streaming video understanding. The method contains three main components. First, Temporal-aware Adaptive K-Means Event Segmentation (TAKC) partitions incoming video streams into event-level units using frame representations, K-means clustering, silhouette-based selection of the number of clusters, and temporal boundary localization. Second, Temporal Segmentation and Aligned Frame Compression (TSAFM) identifies temporally changing frames and preserves them while merging relatively stable frames to reduce visual redundancy. Third, EventMem maintains complementary long-term event memory and short-term residual memory: compressed historical events provide long-range context, while recent and under-represented raw visual evidence is retained to preserve fine-grained information. The framework is training-free and is evaluated with LongVA and Qwen2.5-VL on StreamingBench Real-Time, StreamBench, and OVO-Bench Real-Time. The reported results show improvements over the corresponding backbones, particularly on Object Search, Long-term Memory, and Short-term Memory tasks.

Critical strengths

The paper addresses an important and practical problem in streaming video understanding. Maintaining useful long-term information while controlling the continuously growing visual context is a central challenge for online video MLLMs. The distinction between compressed historical event memory and fine-grained residual memory is intuitive and well motivated. The complete framework produces meaningful improvements without additional training. In particular, the improvements on memory-oriented tasks such as Object Search and Long-term Memory suggest that explicitly organizing historical information can be useful. The component ablations also indicate that the proposed long-/short-term memory decomposition contributes to downstream performance.

Critical weaknesses
  1. The comparison with existing training-free streaming methods is not sufficiently controlled. Several relevant methods use the same Qwen2.5-VL backbone, but the paper does not provide same-backbone and same-memory-budget comparisons, making it difficult to judge whether EventMem itself is better.

  2. The technical novelty is somewhat limited. TAKC mainly relies on K-means clustering and heuristic boundary selection, while TSAFM uses temporal change detection and token merging. The overall framework is reasonable, but many components are relatively incremental.

  3. Some key claims are not directly validated. The paper does not directly evaluate event segmentation quality or provide sufficient memory/latency efficiency analysis, although event discovery and efficient streaming memory are central motivations of the method.T

Questions for authors
  1. Could the authors provide controlled comparisons with recent training-free streaming memory methods that also use Qwen2.5-VL-7B, such as HERMES, FluxMem, and OASIS?

  2. Could the authors compare EventMem with a simple recent-frame baseline such as SimpleStream under the same Qwen2.5-VL-7B backbone and the same visual-token budget?

  3. Could the authors provide latency, peak memory usage, and average retained visual-token statistics for EventMem and competing methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.