acceptodds
Under review as a conference paper at ICLR 2027

Towards a Dynamic and Fixed-budget Memory Bank for Streaming Video Understanding

Abstract

Currently, streaming video understanding is still a daunting task for existing Multimodal Large Language Models (MLLMs). Its difficulties not only lie in handling the ever-increasing video frames, but also in the unpredictability of future video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget memory bank, and propose a novel and training-free approach termed CausalMem. CausalMem targets at constructing a dynamic visual memory update mechanism, thereby maximizing the amount of information in streaming video within a limited memory space, much like human brain. In practice, CausalMem estimates the redundancy of visual tokens and updates the memory bank via carefully constructed online semantic basis, which models the principal semantics of video streams. To validate CausalMem, we apply it to three representative MLLMs, namely LLaVA-OneVision, Qwen2.5-VL and Qwen3-VL respectively, and conduct extensive experiments on both streaming and offlineEvaluated under streaming protocol. video understanding benchmarks. The experimental results not only show the great advantages than existing methods using streaming or offline settings, e.g., and average accuracy gains respectively, but also witness the superior semantic preservation for streaming videos, e.g., using 12 token budgets to memorize hour-long streaming videos, which achieves more than 20 visual token compression ratio and only occupies about 82 MB storage. Our code is given in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fixed-budget memory, Streaming video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JQGROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CausalMem is a training-free visual token memory mechanism for streaming video understanding. It maintains a bounded visual memory and an online orthonormal basis intended to represent the principal semantics of previously observed frames. The method does not require future frames or advance knowledge of user questions. Experiments on three MLLM backbones and streaming and offline video benchmarks report improvements over the underlying models and several compression baselines, together with reduced inference time and GPU memory usage.

Critical strengths
  1. CausalMem maintains bounded visual memory without additional training or advance knowledge of user queries. Its causal, pre-LLM compression design has practical value for resource-constrained streaming video understanding.

  2. The method improves aggregate performance across three MLLM backbones and remains competitive with offline compression methods despite its causal constraint. Its improvement on MLVU’s multi-detail category suggests an ability to preserve useful historical information.

Critical weaknesses
  1. There has already some works on fixed-budget and query-independent streaming memory. The paper lacks the controlled comparisons against alternative semantic similarity, representativeness, or diversity-based selection strategies.

  2. The evaluation does not establish whether pruning visually redundant tokens preserves the counts, order, and duration of repeated actions, limiting the evidence for its benefits to temporal understanding.

Questions for authors
  1. What evidence shows that removing low-residual tokens preserves action counts, event order, and duration for repeated actions? Can the authors distinguish compression-induced information loss from limitations caused by sampling at 0.5/1 fps?

  2. Why does the final configuration score 65.7 on OVO in Table 5 but 67.0 in Tables 1 and 3?

Reviewer_gb8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CausalMem, a training-free approach for streaming video understanding that maintains a dynamic visual memory bank under a fixed token budget. It estimates token redundancy and updates the memory using online semantic bases to preserve relevant information from incoming frames. Experiments on three MLLMs show improvements across streaming and offline video benchmarks, with substantial visual token compression for long videos.

Critical strengths
  • CausalMem addresses the practical challenge of streaming video understanding with a fixed memory budget, without requiring additional model training.

  • The method is evaluated on three representative MLLMs and both streaming and offline benchmarks. The reported results suggest that it can preserve useful visual information while substantially reducing memory usage.

Critical weaknesses
  • Related approaches, such as STC-Pruner and FluxMem, already explore online visual token compression and memory management. The paper lacks direct comparisons with these methods and simple historical feature summarization baselines, making the additional value of the proposed dynamic semantic bases unclear.

  • Although CausalMem and StreamingTOM are both reported with a 12K-token budget in Table 1, StreamingTOM may use substantially fewer tokens when the observed video history is short. VisionZip also does not appear to use a strictly matched input budget. Without actual token usage statistics and budget-matched comparisons, the reported performance gains may partly result from differences in input information.

  • In Table 5, the recency-only variant achieves 74.0 on StreamingBench and 65.3 on OVO, close to the full method's 74.3 and 65.7. These small differences raise questions about the necessity of the proposed semantic bases in streaming scenarios. Moreover, the paper lacks systematic comparisons with fixed semantic bases, historical averages, or similarity-based selection under matched memory budgets.

  • The OVO score of the full method is 67.0 in Table 1 but 65.7 in Table 5. The paper does not clearly explain this discrepancy, making the experimental results difficult to interpret.

Questions for authors
  1. How does CausalMem compare with STC-Pruner, FluxMem, and simple historical feature summarization methods under the same memory budget?

  2. Can the authors report actual input token usage across evaluated examples and provide strictly budget-matched comparisons with StreamingTOM and VisionZip?

  3. Given the small gains of the full method over the recency-only variant on StreamingBench and OVO, what is the specific contribution of the dynamic semantic bases? Can the authors compare them with fixed semantic bases and simpler history-based selection methods under matched budgets?

  4. Why does the full method achieve an OVO score of 67.0 in Table 1 but 65.7 in Table 5? Are these results obtained under different experimental settings?

Reviewer_NHoFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to construct a dynamic but fixed-budget memory bank and a training-free memory updating mechanism CausalMem. CausalMem estimates token redundancy and construct online semantic basis that models the principal semantics of video streams. CausalMem is applied to 3 MLLMs: LLaVA-OneVision, Qwen2.5-VL and Qwen3-VL to show better performance and semantic preservation for streaming videos and token efficiency.

Critical strengths

The method is simple and principled. Training-free and generalizable over multiple open source MLLMs. Strong efficiency.

Critical weaknesses

Is the method developed on specified models and benchmarks? can it generalize to hold out MLLMs and benchmarks ? Limited gains from the semantic mechanism: in Table 5(d) improvements are small. Is long-term forgetting measured somewhere?

Questions for authors

Is the method developed on specified models and benchmarks? can it generalize to hold out MLLMs and benchmarks ?

1 comment

Dear Reviewer NHoF,

Thanks again for your great effort in reviewing our paper.

In our rebuttal, we have carefully responded to all your concerns, such as hyperparameters generalization across models and benchmarks (Q1R1), the contribution of the semantic mechanism (Q2R2), and long-term memory retention (Q3R3):

  1. hyperparameters Generalization across models and benchmarks (Response 1): We clarified that the key hyperparameters (q=64, m=8, and γ=0.9) are shared across all evaluated MLLMs and benchmarks, supporting applicability across different settings without tuning these parameters on different benchmarks and models.

  2. Contribution of the semantic mechanism (Response 2): We explained how temporal recency bias affects the ablation results and provided additional comparisons with simple memory construction methods under the same token budget. CausalMem consistently outperforms random, similarity-based, and attention-based pruning on MLVU, OVOBench, and VideoMME.

  3. Long-term memory retention (Response 3): We report OVOBench’s backward-tracing task and the splited results on LVBench, which splites questions by the timestamps of visual relevant evidence appearing in hour-long videos. The results show that CausalMem preserves information appearing early in long videos, supporting long-term retention instead of only retenting recent visual information.

With the discussion period drawing to a close, we look forward to hearing from you and will be very delighted to answer any further questions you may have.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.