acceptodds
Withdrawn from ICLR 2027

Long-Term Visual Understanding with Latent Token Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While Large Multimodal Models (LMMs) have demonstrated impressive capabilities in reasoning over static images, they face severe scalability bottlenecks when processing extensive visual streams, such as long-form videos or multi-page documents. Prevailing approaches typically resort to raw token concatenation, which incurs quadratic computational costs, or aggressive stateless pruning, which inevitably sacrifices fine-grained semantic details. To address this, we introduce MemVLM, a novel framework that treats visual history as an online information processing problem. Instead of buffering an ever-growing queue of tokens, MemVLM maintains a fixed-size latent belief state, acting as a sufficient statistic for the visual history. Our architecture employs a dual-head projection to decouple immediate visual grounding from long-term memory updates, utilizing a Gated Recurrent Memory Update (GRMU) mechanism to distill high-entropy evidence into the latent state. Guided by a "Minimal Write Principle," the model learns to selectively update memory only when necessary, effectively transforming the transformer into a state-space model with memory complexity. Empirical evaluations on comprehensive long-context benchmarks demonstrate that MemVLM significantly outperforms state-of-the-art token pruning methods, enabling the processing of thousands of frames without "lost-in-the-middle" degradation or memory explosion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Streaming VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aTUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper bolts a small recurrent memory onto an off-the-shelf VLM so it can watch a long video without the context growing. Every frame goes through frozen CLIP, gets squeezed to a few tokens, and is written into a fixed bank of memory vectors through a GRU-like gate. When a question arrives the LLM sees only the current frame and that bank, inserted as KV prefixes in a handful of layers. An L1 penalty on the gates makes most frames write nothing. The LLM is LoRA-tuned on LLaVA-Video-178K with 8-frame truncated BPTT.

The authors report experimental results on 8 video benchmarks at 128, 64 and 32 visual tokens as retention against an "oracle" (91% for MemVLM at 128 tokens, 86.5% for the best pruning baseline), VNBench needle tests (35.4 against 18.1 for 32-frame Video-LLaVA), a dense-event ordering stress test with a capacity sweep, a VisPruner fine-tuned on the same data, a transfer to LLaVA-Video-7B measured against a sliding-window version of that model, a 10,000-frame memory curve, and ablations of the heads, the update rule and the sparsity weight.

Critical strengths
  1. The design is simple and the systems result is real. A gated memory injected as KV prefixes is easy to retrofit onto an existing VLM, it holds at 16.5 GB through 10,000 frames where concatenation dies at about 220, and the paper is upfront that the memory is lossy and shows exactly when (Tables 6 to 8).

  2. The ablations are convincing on their own terms. Swapping GRMU for FIFO, mean pooling, a plain RNN or momentum costs 7 to 14 points, dropping the dual head costs 4, and both extremes of the sparsity weight hurt. The fine-tuned VisPruner control answers the obvious objection that the gain is just extra video data.

Critical weaknesses

1. The closest prior work is missing. Memory-based streaming video LMMs are a well-populated line (MovieChat, MA-LMM, Flash-VStream, VideoLLaMB, VideoStreaming, LongVU, and more recent streaming models), and several of them do more or less what GRMU does: compress each frame, update a recurrent or queue-style memory, feed it to the LLM. None is compared, none is cited. The only "streaming" baseline is SlowFast-LLaVA, which is a training-free sampler, not a memory. So I cannot tell whether MemVLM is new, or whether it beats the methods it most needs to beat. This alone puts me at reject.

2. The headline numbers do not seem to be convicing. Baseline frame counts are not given, so the budget matching may not be matching anything; on VNBench the baseline sees 32 frames and MemVLM sees them all. The oracle row is higher than the backbone's own published results and is not explained. The transfer check is against a handicapped sliding-window variant rather than LLaVA-Video-7B's normal inference, and the 69.7 baseline happens to equal that model's published overall with-subtitles Video-MME score, not a Long-subset score. Add the inconsistent default budget and the Table 1 versus Table 11 mismatch.

Questions for authors
  1. What exactly is the oracle in Table 1: which checkpoint, how many frames, with or without subtitles, fine-tuned on LLaVA-Video-178K or not? Why do its ActivityNet-QA and MSRVTT-QA numbers exceed the Video-LLaVA paper's own?
  2. For every row of Tables 1 and 2, how many frames does the method consume, and is B_vis per frame or per video? On VNBench, how many frames does MemVLM process? Please add a uniform-sampling baseline that sees the same number of frames, pooled to the same token count.
  3. Compare against at least 2 memory-based streaming method (MA-LMM or Flash-VStream) on the same backbone and data, and explain how GRMU differs from their memory updates.
Reviewer_2CnAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the issues of memory explosion caused by concatenating historical tokens and information loss resulting from stateless pruning when large multimodal models process long video streams, this paper proposes the MemVLM framework. This framework uses a budget-constrained dual-head projector to separate real-time visual grounding from long-term memory writing, and employs Gated Recurrent Memory Update (GRMU) and minimum-write regularization to maintain a fixed, compact hidden state. Experimental results show that MemVLM outperforms existing retrieval-based and pruning-based baselines on multiple long-video tasks, and can achieve constant-memory inference of up to 10,000 frames on a single GPU.

Critical strengths

1.Dual-Head Projector: Employs a Read Head to process current visual information, while a Write Head encodes video frames into compact tokens used to update the memory.

2.Gated Recurrent Memory Update (GRMU): Continuously updates a fixed-capacity implicit memory through cross-attention, candidate memory states, and gated interpolation.

3.Minimal Write Regularization: Penalizes the activation magnitude of the memory update gates to minimize the interference of redundant frames with the historical memory.

Critical weaknesses
  1. Why does the score for MemVLM on Video-MME (Long) in Table 1 (45.6) differ from the score obtained under the default 8-frame TBPTT configuration in Table 11 of the appendix (43.6)? Please provide a precise clarification of the mathematical relationship between the visual token budget Bvis∈{128,64,32}B_{\text{vis}} \in \{128, 64, 32\} and the hyperparameters R,B,KR, B, K, and ∣S∣|S|. When Bvis=128B_{\text{vis}} = 128, why does the main text state B=64B = 64 and R=64R = 64, while page 14 of the appendix and Table 8 indicate B=128B = 128? If ∣S∣=7|S| = 7 layers all have memory, is it truly fair to compare it with a baseline (such as VisPruner) where all layers share 128 input tokens?

  2. In Algorithm 1, the Xread,tX_{\text{read},t} values computed in the first T-1 frames are continuously overwritten in line 12 and do not contribute to the memory update. Is this a mistake in the pseudocode, or does the implementation actually recalculate the unnecessary Read Head every frame?

  3. In offline long video Q&A (such as Video-MME), what is the rationale for only using XTreadX_T^{\text{read}} of the last frame vTv_T of the video as the immediate visual token input to the LLM? If a variant with R=0R=0, B=128B=128 (where the entire token budget is allocated to long-term memory) is tested, how does it perform on long-video benchmarks?

  4. How exactly is the memory state Mtl∈RB×DM_t^l \in \mathbb{R}^{B \times D} projected into the KV caches within Transformer layers, and does this projection reuse the frozen WKlW_K^l and WVlW_V^l from the backbone LLM? Which specific Transformer layers are included in the set S\mathcal{S}? How do the position and count of injection layers affect model performance and GPU memory consumption?

  5. The ϵ\epsilon-sufficient statistics is merely proposed as an optimization objective, and no rigorous proof is provided that GRMU can satisfy this property. The drift bound derived in Proposition 3.2 only demonstrates that sparse gating can constrain the variation of memory states, but cannot guarantee the preservation of task-relevant information. Furthermore, Proposition 3.4 rests on strong gradient approximation assumptions. Its conclusions are primarily applicable to inactive memory update phases, and are insufficient to fully account for the memory stability of the model when processing long-term complex visual sequences.

Questions for authors

The authors are encouraged to refer to the Critical Weaknesses section for further details and suggestions for improvement.

Reviewer_y3BPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MemVLM for long video understanding under a fixed visual memory budget. A dual-head projector separates immediate visual grounding from memory writes, while gated recurrent updates maintain compact historical representations that are injected into the LLM as KV prefixes. A Minimal Write regularizer discourages unnecessary updates. Experiments evaluate accuracy, efficiency, and robustness across video understanding benchmarks.

Critical strengths
  1. Practical motivation and intuitive design. The paper tackles the practical challenge of long-video understanding under a fixed memory budget. Decoupling immediate visual grounding from recurrent memory updates yields a clean and intuitive architecture.

  2. Thorough evaluation and limitation analysis. The evaluation spans multiple benchmarks, component ablations, and efficiency tests. Furthermore, experiments across varying event densities, memory capacities, and video lengths clearly characterize the method's boundaries.

Critical weaknesses
  1. The abstract labels 69.7 as a Video-MME (Long) baseline, but this matches LLaVA-Video-7B’s published overall accuracy with subtitles. Please verify whether the reported improvement from 69.7 to 73.1 concerns the overall benchmark or the Long subset. This distinction must be correct in the abstract.

  2. KV Cache injection is central to the method, yet its implementation is missing. How are memory states converted into keys and values, which LLM layers receive them, and how are positional encodings handled?

  3. Memory updates depend sequentially on previous frames. How does this affect training parallelism and inference throughput? Please clarify what can run in parallel and whether the reported throughput includes the full memory-update pipeline.

  4. How many frames does each method process on each benchmark? Input frame counts, sampling rates, and maximum frame limits are missing. Without these details, the results cannot be independently reproduced, and comparable access to visual evidence cannot be verified.

Questions for authors

Can a single trained checkpoint support different read budgets RR, memory capacities BB, and write budgets KK at inference time? How are the learned queries and memory slots adjusted?

Were the 8-frame and 32-frame Video-LLaVA baselines on VNBench fine-tuned on the same data as MemVLM? If not, how much of the improvement comes from additional training rather than the memory mechanism?

The contrast with token pruning is overstated. From my perspective, fixed-capacity latent memory is also a lossy compression mechanism, particularly as task-relevant evidence accumulates over long streams.

Reviewer_JxjeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemVLM, a fixed-budget latent visual memory framework that formulates long-term visual stream understanding as online state estimation. MemVLM consists of three main components: (1) a Budgeted Dual-Head Visual Projector using Perceiver-style cross-attention resamplers to separate immediate visual grounding (RR read tokens) from long-term memory distillation (KK write tokens); (2) a Gated Recurrent Memory Update (GRMU) module that maintains layer-wise latent states Mtl∈RB×DM_t^l \in \mathbb{R}^{B \times D} via cross-attention and GRU-style gated interpolation, injecting the final states as read-only KV-cache prefixes during inference; and (3) a Minimal Write objective that applies ℓ1\ell_1 regularization on gate activations to encourage sparse, event-triggered updates. The authors evaluate MemVLM on Video-LLaVA-7B, LLaVA-1.5-7B, and LLaVA-Video-7B across standard video QA benchmarks, VNBench, and synthetic dense-event stress tests

Critical strengths
  1. Constant-Memory Streaming Scalability: By maintaining a fixed-size latent state injected as a KV prefix rather than concatenating historical visual tokens, MemVLM achieves O(1)\mathcal{O}(1) memory scaling with stream length, sustaining ~16.5 GB GPU memory and ~36 FPS up to 10,000 frames on a single A100-80GB where full concatenation runs out of memory at ~220 frames.
  2. Insightful Stress Tests on Failure Boundaries: The evaluation goes beyond standard accuracy tables by probing temporal ordering under varying event densities, memory capacities (B∈{32,64,128}B \in \{32, 64, 128\}), and TBPTT truncation windows, clearly illustrating the capacity-demand trade-offs of fixed-size latent memory.
Critical weaknesses
  1. Incremental Architecture, Trivial Theory, and Missing Memory-Based Baselines:

    • Incremental Design & Trivial Proofs: Architecturally, the dual-head projector is a pair of standard Perceiver resamplers, and GRMU combines standard multi-head cross-attention with a classic GRU convex gating update (Mtl=(1−gtl)⊙Mt−1l+gtl⊙M~tlM_t^l = (1-g_t^l)\odot M_{t-1}^l + g_t^l\odot \tilde{M}_t^l). Furthermore, Propositions 3.2 and 3.4 are elementary algebraic consequences of assuming unconditionally bounded updates (Assumption 3.1) and ignoring gate gradients (Assumption 3.3), merely restating basic properties of GRU forget gates.
    • Missing Streaming/Memory Baselines: Despite focusing on recurrent visual memory, Table 1 only includes SlowFast-LLaVA (a training-free two-stream pooling method) as a "streaming" baseline, omitting established memory-augmented and streaming VideoLLMs such as MovieChat, MA-LMM, Flash-VStream, LongVU, and LongLive.
  2. Unfair Main-Table Comparisons and Numerical Inconsistencies:

    • Unfair Training Setup in Table 1: MemVLM is trained with LoRA and end-to-end memory modules on LLaVA-Video-178K. Cross-referencing Table 1 with Table 9 reveals that VisPruner in Table 1 (49.8 on Video-MME All, 49.5 on ActivityNet-QA) is actually the training-free version, whereas VisPruner + matched finetuning improves to 50.8 and 50.6 in Table 9. Mixing fine-tuned MemVLM with training-free pruning baselines (SparseVLM, PruMerge+, VisPruner) in Table 1 without clear labeling is misleading.
    • Conflicting and Implausible Metrics: (i) Under the default 128-token budget and 8-frame TBPTT setting, MemVLM's Video-MME (Long) score is reported as 45.6 in Table 1 but 43.6 in Table 11. (ii) In the Abstract, Section 4.3, and Table 10, LLaVA-Video-7B + Sliding Window and + MemVLM reportedly achieve 69.7 and 73.1 on Video-MME (Long), which is drastically higher than the official uncompressed 64-frame LLaVA-Video-7B performance on Video-MME Long (~53.4%, or ~64.3% Overall).
  3. Ambiguous KV-Prefix Injection, Budget Accounting, and Redundant Read-Head Execution:

    • Unspecified KV Mapping & Layer Selection: The paper defines Btot=∣S∣BB_{\text{tot}} = |\mathcal{S}|B with default B=64,R=64,K=32B=64, R=64, K=32, yet never specifies which or how many layers are in S\mathcal{S} (Fig. 1 shows *7 Layers), nor how a single state tensor Mtl∈RB×DM_t^l \in \mathbb{R}^{B \times D} is mapped into separate Key and Value tensors inside the LLM's KV cache. It is also unclear whether Bvis=128\mathcal{B}_{\text{vis}}=128 means B+R=64+64B+R=64+64 or B=128B=128 (as written in Line 750).
    • Wasted Read-Head Computation: In Algorithm 1 (Lines 4 and 12), ReadHead(H_t, R) is computed for every frame t=1…Tt=1\dots T, only to be overwritten at each step so that solely the final frame XTreadX_T^{\text{read}} is passed to LLM.generate. Computing read tokens for frames 1…T−11\dots T-1 is redundant, and relying solely on the last frame TT (often black credits in offline videos) for immediate grounding is questionable.
Questions for authors

Please see Critical Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.