acceptodds
Withdrawn from ICLR 2027

HippoStream: Hippocampal Memory for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The hippocampus is associated with the rapid encoding of recent experience into a compact representation that other regions can later draw on. Streaming Video LLMs face the same problem with no comparable mechanism: they either keep a sliding window of recent frames and forget the past, or grow a key–value cache that scales with stream length and breaks on long videos. HippoStream is a small parametric token bank that the inference loop rewrites with functional gradient steps per chunk while the Video LLM trunk stays frozen. The bank plays the role of hippocampus, the frozen trunk the role of neocortex. Its initial state, writer head, and read projection are meta-learned offline; per-chunk cost is independent of stream length, and the bank's footprint stays constant across videos of any length. On OVO-Bench with Qwen3-VL-8B, HippoStream improves the Backward-summary cluster (Action-State Inference, Hold-out reasoning) by over the recent-window baseline. The gain traces to a single design choice in the WRITE step: anchoring the bank's output positions to each chunk's first vision token preserves the event-specific evidence that backward retrieval needs. Combined with offline meta-learning of the slow weights, this gives a streaming Video LLM reliable access to past events long after the recent window has scrolled by. We release code, meta-trained checkpoints, and the training-data replay buffer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video LLMs, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DQTvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HIPPOSTREAM, a fixed-size bank of token-shaped states for streaming video question answering. Each one-second chunk produces one frame and one reconstruction-gradient update to the bank, while first-order meta-training learns its initialization and write/read interfaces with Qwen3-VL frozen. Evaluations on OVO-Bench and VSI-SUPER-Recall show gains on coarse backward-summary tasks, alongside losses on detailed episodic recall and several current-frame tasks.

Critical strengths
  • Updating a small token bank by gradients provides a distinct fixed-state alternative to caches and retrieval stores whose storage grows with the stream. The persistent state remains M×dM \times d, and the cost of each WRITE does not depend on how much earlier video has already been processed.

  • Table 14 reports a 6.76-point improvement on Action-State Inference over the no-memory baseline. This is concrete evidence that four persistent tokens can retain useful coarse information about earlier events.

Critical weaknesses
  • The WRITE objective does not directly establish that the four-token bank preserves evidence needed by future questions. Equations (1)–(2) repeatedly move the bank toward a pooled or first-position representation of each chunk, and first-order MAML prevents the answer loss from directly training WhW_h. The target ablations compare compression heuristics, but do not show whether answer-critical details survive repeated overwriting.

  • The empirical gains are concentrated on coarse summary memory rather than detailed episodic recall. Table 14 reports +6.76+6.76 on ASI but −1.68-1.68 on EPM, while every HLD answer is Unable to answer and the corrected-prompt gain in Table 13 is only +1.61+1.61. The gradient bank also falls from 56.7 at 10 minutes to 31.7, 28.3, and 15.0 at 30, 60, and 120 minutes in Table 16, so the broader claim of reliable access to old events is not yet supported.

  • The properly meta-trained memory does not provide a meaningful aggregate gain in Table 9. The no-memory baseline scores 58.52, compared with 58.28, 58.24, and 58.53 for the first-position, mean-pooled, and attention-pooled d=3584d=3584 variants. These results support a redistribution from real-time and detailed-recall tasks toward coarse backward-summary tasks, while the headline shape-skip rows use random slow components and therefore cannot establish that offline meta-learning is responsible.

  • Constant state size does not imply low inference cost. Every one-second chunk requires a frozen-LLM forward and backward pass, and Appendix G reports 5–8 hours for a full memory-on evaluation and up to approximately 30 minutes for one Forward query. Without matched no-memory latency, streaming throughput, peak GPU memory, and a forward-only recurrent or bounded-memory baseline, it is difficult to assess whether the storage saving justifies the additional compute.

Questions for authors
  1. It would be helpful to evaluate whether the bank preserves answer-critical details using multiple questions per video, including questions about low-salience past events. Comparing the current reconstruction-trained writer with one that receives answer-loss gradients would further clarify whether the WRITE objective retains more than dominant chunk content.

  2. Please report a full-1640 comparison under one corrected prompt and scorer between random and properly meta-trained d=3584d=3584 slow components, with separate ASI, EPM, HLD, real-time, and overall results. This would show whether MAML adds value beyond the fast gradient update itself.

  3. A matched systems comparison would make the fixed-memory trade-off clearer. Reporting latency per chunk, sustainable streaming FPS, query latency, throughput, and peak GPU memory for no memory, HIPPOSTREAM, a forward-only recurrent memory, and a bounded retrieval baseline would be particularly informative.

Reviewer_wMgLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HippoStream for streaming video understanding with a frozen Video LLM and a small, updatable bank of memory tokens. The key idea is to write incoming video chunks into the bank through test-time gradient updates, then read the projected bank together with a recent visual window to answer questions. The proposed training procedure learns the bank initialization, writer head, and read projection offline. The evaluation mainly uses OVO-Bench, which covers backward tracing, real-time perception, and forward reasoning, with ablations of the WRITE target, bank size, update gating, read placement, and meta-training, etc. Further experiments examine recent-window and retrieval baselines across backbones and on StreamingBench, as well as longer-term recall on VSI-SUPER-Recall using a caption-based bank and external retrieval.

Critical strengths
  1. I think a small memory that can be updated online while keeping the Video LLM frozen is a useful direction. The method is simple, and the fixed bank size gives a clear storage advantage over retaining the full visual history.

  2. There is a substantial amount of experiments conducted around the proposed design, which provides valuable insights for the method and limitations.

  3. Overall, the main idea and the WRITE/READ workflow are clearly presented.

Critical weaknesses
  1. Novelty. The method is closely related to online gradient descent updating methods such as TTT and Titans, with the difference that the fast state is a set of tokens rather than network weights. The paper mentioned GradMem, PERK, and video-SALMONN S. I would expect more comparison with those methods.

  2. scope. Most of the experiments stay within variations of the proposed pipeline. The main experiments are concentrated on Qwen3-VL and OVO-Bench. The longer-video extension focuses on four inserted object occurrences, and its strongest results come from external retrieval (storage that grows with video length). I would like to see stronger evidence that the bank itself can handle more diverse long-term questions, rather than just tuning its current configuration.

  3. Meta-learning. If my understanding of Table 9 and Appendix S is correct, the 8B results do not clearly establish the benefit of meta-learning: some runs use random slow components, and training them at the correct dimension adds no clear overall gain. Hence, I am not sure how much of the headline improvement comes from meta-learning. Figure 2 (b) also evaluates the two WRITE targets on different subsets, so that comparison alone does not isolate the effect of the target.

  4. Backward-summary gain. I also think the backward-summary gain needs a more careful interpretation. Appendix K states that all “Unable to answer” is the correct answer for all HLD questions, so improvement there alone does not tell how much past information is retained. Table 1 reports +6.45 on HLD, while Table 13 reports +1.61 for the vanilla M=4 writer with the corrected prompt. I would like to understand how these settings relate, and whether the ASI+HLD headline gain remains with the corrected prompt.

  5. I find some of the explanations not very conclusive. For example, (1) varying the bank size does not by itself establish attention bandwidth as the main bottleneck, and Table 11 pools very small samples from different writers. (2) Direct attention measurements or controlled changes to the historical evidence would tell us more about what the bank actually stores and uses. (3) Similarly, the QA results in Table 16 seem insufficient to describe the write as “lossless.”

Questions for authors

I hope the authors can respond to the weaknesses I listed above. Besides,

  1. Please clarify the differences between the full-1640 results in Tables 2 and 9, as well as the 80-video and 200-video meta-training configurations.
Reviewer_gHWEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

HippoStream gives a streaming video LLM a memory of fixed size. They take frozen LLM and use memory of just four tokens. This design is very loosely inspired by the hippocampus of the brain. The idea is the memory never grows. They credit the gain to how the write step builds its target from each chunk's first vision token.

Critical strengths
  1. The idea is simple and cheap. There is no growing memory costs.

  2. The authors are very candid the appendix lists failures and they admit the gain is within noise.

Critical weaknesses
  1. The paper's own words contradict its main claim. The abstract credits "offline meta-learning of the slow weights" for giving the model "reliable access to past events." Section 2.4 goes further: "Without a meta-trained writer, the four bank positions read as noise to the LLM." Yet the headline 8B result uses no meta-learning at all: "the 8B configuration runs with random-initialised slow components" (Section 3.1). When the authors do meta-train at 8B, there is no gain (58.24–58.53 vs. a 58.52 baseline, Table 9). Appendix D admits the meta-learned warm start "does not compound the lift we hoped for." So by the paper's own account, the headline result comes from the very setup the authors say should read as noise.

  2. The paper is unfinished. Several results are marked as in progress, including an empty row in Table 12, "Table 9 will be updated" (Appendix F), and "will be replaced with the final-N values at submission time" (Appendix J).

  3. The tables are not formatted to within the paper limits. The appendix includes details that may compromise anonymity, such as the compute cluster name (Bridges-2, Appendix G)

Questions for authors

Section 2.4 states that "without a meta-trained writer, the four bank positions read as noise to the LLM," yet Section 3.1 states that the headline 8B result "runs with random-initialised slow components," and the properly meta-trained 8B runs show no gain (58.24–58.53 vs. a 58.52 baseline, Table 9). If the headline result comes from a setup the authors themselves describe as noise, what evidence supports the abstract's claim that meta-learning gives the model "reliable access to past events"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.