- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CausalMem is a training-free visual token memory mechanism for streaming video understanding. It maintains a bounded visual memory and an online orthonormal basis intended to represent the principal semantics of previously observed frames. The method does not require future frames or advance knowledge of user questions. Experiments on three MLLM backbones and streaming and offline video benchmarks report improvements over the underlying models and several compression baselines, together with reduced inference time and GPU memory usage.
Critical strengths
-
CausalMem maintains bounded visual memory without additional training or advance knowledge of user queries. Its causal, pre-LLM compression design has practical value for resource-constrained streaming video understanding.
-
The method improves aggregate performance across three MLLM backbones and remains competitive with offline compression methods despite its causal constraint. Its improvement on MLVU’s multi-detail category suggests an ability to preserve useful historical information.
Critical weaknesses
-
There has already some works on fixed-budget and query-independent streaming memory. The paper lacks the controlled comparisons against alternative semantic similarity, representativeness, or diversity-based selection strategies.
-
The evaluation does not establish whether pruning visually redundant tokens preserves the counts, order, and duration of repeated actions, limiting the evidence for its benefits to temporal understanding.
Questions for authors
-
What evidence shows that removing low-residual tokens preserves action counts, event order, and duration for repeated actions? Can the authors distinguish compression-induced information loss from limitations caused by sampling at 0.5/1 fps?
-
Why does the final configuration score 65.7 on OVO in Table 5 but 67.0 in Tables 1 and 3?
Dear Reviewer NHoF,
Thanks again for your great effort in reviewing our paper.
In our rebuttal, we have carefully responded to all your concerns, such as hyperparameters generalization across models and benchmarks (Q1R1), the contribution of the semantic mechanism (Q2R2), and long-term memory retention (Q3R3):
hyperparameters Generalization across models and benchmarks (Response 1): We clarified that the key hyperparameters (q=64, m=8, and γ=0.9) are shared across all evaluated MLLMs and benchmarks, supporting applicability across different settings without tuning these parameters on different benchmarks and models.
Contribution of the semantic mechanism (Response 2): We explained how temporal recency bias affects the ablation results and provided additional comparisons with simple memory construction methods under the same token budget. CausalMem consistently outperforms random, similarity-based, and attention-based pruning on MLVU, OVOBench, and VideoMME.
Long-term memory retention (Response 3): We report OVOBench’s backward-tracing task and the splited results on LVBench, which splites questions by the timestamps of visual relevant evidence appearing in hour-long videos. The results show that CausalMem preserves information appearing early in long videos, supporting long-term retention instead of only retenting recent visual information.
With the discussion period drawing to a close, we look forward to hearing from you and will be very delighted to answer any further questions you may have.
Best regards,
The authors