- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper bolts a small recurrent memory onto an off-the-shelf VLM so it can watch a long video without the context growing. Every frame goes through frozen CLIP, gets squeezed to a few tokens, and is written into a fixed bank of memory vectors through a GRU-like gate. When a question arrives the LLM sees only the current frame and that bank, inserted as KV prefixes in a handful of layers. An L1 penalty on the gates makes most frames write nothing. The LLM is LoRA-tuned on LLaVA-Video-178K with 8-frame truncated BPTT.
The authors report experimental results on 8 video benchmarks at 128, 64 and 32 visual tokens as retention against an "oracle" (91% for MemVLM at 128 tokens, 86.5% for the best pruning baseline), VNBench needle tests (35.4 against 18.1 for 32-frame Video-LLaVA), a dense-event ordering stress test with a capacity sweep, a VisPruner fine-tuned on the same data, a transfer to LLaVA-Video-7B measured against a sliding-window version of that model, a 10,000-frame memory curve, and ablations of the heads, the update rule and the sparsity weight.
Critical strengths
-
The design is simple and the systems result is real. A gated memory injected as KV prefixes is easy to retrofit onto an existing VLM, it holds at 16.5 GB through 10,000 frames where concatenation dies at about 220, and the paper is upfront that the memory is lossy and shows exactly when (Tables 6 to 8).
-
The ablations are convincing on their own terms. Swapping GRMU for FIFO, mean pooling, a plain RNN or momentum costs 7 to 14 points, dropping the dual head costs 4, and both extremes of the sparsity weight hurt. The fine-tuned VisPruner control answers the obvious objection that the gain is just extra video data.
Critical weaknesses
1. The closest prior work is missing. Memory-based streaming video LMMs are a well-populated line (MovieChat, MA-LMM, Flash-VStream, VideoLLaMB, VideoStreaming, LongVU, and more recent streaming models), and several of them do more or less what GRMU does: compress each frame, update a recurrent or queue-style memory, feed it to the LLM. None is compared, none is cited. The only "streaming" baseline is SlowFast-LLaVA, which is a training-free sampler, not a memory. So I cannot tell whether MemVLM is new, or whether it beats the methods it most needs to beat. This alone puts me at reject.
2. The headline numbers do not seem to be convicing. Baseline frame counts are not given, so the budget matching may not be matching anything; on VNBench the baseline sees 32 frames and MemVLM sees them all. The oracle row is higher than the backbone's own published results and is not explained. The transfer check is against a handicapped sliding-window variant rather than LLaVA-Video-7B's normal inference, and the 69.7 baseline happens to equal that model's published overall with-subtitles Video-MME score, not a Long-subset score. Add the inconsistent default budget and the Table 1 versus Table 11 mismatch.
Questions for authors
- What exactly is the oracle in Table 1: which checkpoint, how many frames, with or without subtitles, fine-tuned on LLaVA-Video-178K or not? Why do its ActivityNet-QA and MSRVTT-QA numbers exceed the Video-LLaVA paper's own?
- For every row of Tables 1 and 2, how many frames does the method consume, and is B_vis per frame or per video? On VNBench, how many frames does MemVLM process? Please add a uniform-sampling baseline that sees the same number of frames, pooled to the same token count.
- Compare against at least 2 memory-based streaming method (MA-LMM or Flash-VStream) on the same backbone and data, and explain how GRMU differs from their memory updates.