acceptodds
Under review as a conference paper at ICLR 2027

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Abstract

Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact ≈256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks, in offline and streaming end-of-stream settings, macro-average accuracy improves over frozen baselines at every evaluated decoder budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. The added parameters are ≈0.24% of the backbone parameter count, at 0.03 GiB of peak GPU memory overhead. Code is available at https://anonymous.4open.science/r/PReM.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video understanding, vision-language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yv5DOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PReM, a framework for augmenting frozen vision-language models with recurrent memory for long-video understanding. During query-independent video ingestion, a gated associative update compresses frame features into a fixed-size set of memory matrices. At question-answering time, question-conditioned memory reads steer the keys and values of existing non-visual prefix tokens during prefill, without introducing additional memory tokens.

Critical strengths
  1. The memory-to-decoder interface is a useful and clearly motivated design. PReM separates query-independent video ingestion from question-conditioned memory access. Steering existing prefix K/V representations provides a practical way to expose recurrent memory to a frozen decoder without expanding the prompt with memory tokens.
  2. The paper makes a meaningful effort to isolate the proposed interface. The controlled comparison includes a Token-Readout baseline that uses the same recurrent writer, ingested stream, and trainable parameter budget.
  3. Several practical details are reported transparently. The paper documents the training subset, identifier-based video decontamination, state size, trainable parameter counts, and separate ingestion and answering costs.
Critical weaknesses
  1. The configuration associated with a central result is inconsistent across the manuscript. Appendix A.4 explicitly states that both Token-Readout and PReM use K=1K=1 in Table 2. However, all six PReM benchmark scores in Table 2 exactly match the streaming B=16B=16 results in Table 5, while Appendix A.6 states that the main checkpoints use K=4K=4. These settings also imply different recurrent-state capacities. The manuscript does not establish whether the two tables reuse the same checkpoint, whether one configuration description is incorrect, or whether distinct models independently produced the same rounded results. This ambiguity affects the interpretation of both the controlled comparison and the main results and should be resolved with an explicit checkpoint-to-configuration mapping.
  2. The experiments establish performance on sparsely sampled long videos more clearly than retention over long recurrent streams. Evaluation uniformly subsamples long videos to at most 240 frames and asks questions only after ingestion finishes. Thus, a video lasting over an hour does not necessarily produce more recurrent updates than a much shorter video.
  3. The efficiency advantage needs to be qualified by workload and latency. The small additional GPU memory footprint is well supported, but the measured runtime overhead is appreciable.
Questions for authors
  1. Can Figure 5 be repeated with exactly the same decoder frame indices for Base and PReM? Do the larger gains on longer videos persist?
  2. How does accuracy change when the recurrent stream substantially exceeds 240 frames? Can the authors evaluate retention as a function of evidence age and intervening distractor frames?
  3. How does the proposed writer compare with temporal pooling, an exponential moving average, or a simpler associative update when all use the same prefix-steering readout? How sensitive is Token-Readout to token count and insertion position?
Reviewer_HgvxOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces Prefix-Steered Recurrent Memory for long video understanding. The author proposes a query agnostic writer, which accepts the video stream and encode it into a compact memory. Then for each question, the reader retrieves useful information from the memory and use it to update the keys and values in VLM. The author conducted sufficient experiments to demonstrate the performance and necessity of each component. My main concern is in the presentation, as some parts are hard to understand and less motivated.

Critical strengths
  1. The idea of forming a memory in query-agnostic way and use reader to retrieve is novel, because it improves efficiency while maintaining important evidence.
  2. The author conducted sufficient experiments, which includes testing on variety number of datasets, and through ablation. Results are good.
  3. The paper provides some good qualitative results (e.g., Figure 8).
Critical weaknesses

From the ablation studies and figures in the paper, I believe the method is reasonable. However, the presentation of this paper needs to be improved for better understanding. See questions.

Questions for authors
  1. Terms used in method part should be explained clearly, e.g., address spaces, content spaces, slot matrix, etc. Although there is formular definition, readers will feel confused about what each term is used for.
  2. I'd like to see an updated table of symbols in the method part (like kt,mk_{t,m}, et,me_{t,m}), when they are used, and what do they represent. Currently some formulas lack motivation. Table 12 shows it, but does not explain clearly. For example, what is reconstruction residual and what is it used for. For some symbols, you may explain why it is calculated this way.

I would like to increase my rating to 4 is presentation issue is addressed.

Reviewer_WpGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PREM to improve long-video question answering with frozen vision-language models under a limited decoder visual-input budget. It retains a visual buffer of B frames while writing spatially mean-pooled features from additional sampled frames into a multi-slot recurrent associative memory. Writing is question-independent. At question-answering time, a question-conditioned readout modulates the K/V representations of existing non-visual prompt positions during prefill. The design introduces no additional memory tokens and allows the memory of a video to be reused across questions.

The authors evaluate six video-understanding benchmarks, three frozen backbones, and both offline and streaming end-of-stream protocols. For Qwen2.5-VL-3B at B=16 in the offline setting, macro-average accuracy increases from 50.39% to 53.45%, a gain of 3.06 percentage points. Across three training seeds, the mean gain is 2.99 percentage points with a standard deviation of 0.06. The paper also reports component ablations, same-backbone interface comparisons, and memory and latency measurements. Evaluation uses at most 240 sampled write frames and one-token constrained decoding for multiple-choice answers.

Critical strengths
  1. Decoupling memory ingestion from question answering provides a useful interface. Video memory can be constructed before a question arrives and accessed without additional memory tokens or recurrent updates during generation. Table 7 reports that, at B=16, answering latency increases from 0.30 s to 0.38 s and peak GPU memory increases by approximately 0.03 GiB. Total mean latency nevertheless increases from 5.94 s to 8.19 s. The practical value is therefore the compact state and small incremental cost of repeated questions.

  2. The reported gains span multiple backbones and budgets, with supporting stability and component analyses. Tables 5 and 6 report higher macro-average accuracy than the paired baseline at every evaluated budget for all three backbones. Table 11 supports the training stability of the main B=16 offline gain. In Table 8, full K/V modulation outperforms K-only or V-only modulation, and removing each write-control factor reduces macro-average accuracy. These results support the design's usefulness within the evaluated protocols.

Critical weaknesses
  1. The duration analysis confounds memory effects with buffer sampling, and the retention evaluation has a limited horizon. Appendix B.1 states that Figure 5 compares a uniformly sampled Base buffer with a reservoir-buffer PREM run. Consequently, the gains on longer videos cannot be attributed entirely to recurrent memory. Appendix A.8 also caps ingestion at 240 sampled frames regardless of native video duration, so longer videos do not necessarily entail longer recurrent update sequences. The analysis supports accuracy differences under the specified sampling protocols, but does not by itself establish improved retention as the update sequence grows. This limits the evidence for the long-term memory claim.

  2. Important configuration and mechanism descriptions remain inconsistent. Section 4.1 and Appendix A.6 describe the main configuration as K=4, whereas the controlled comparison in Table 2 explicitly uses K=1. Nevertheless, all six PREM benchmark accuracies and the macro-average in Table 2 exactly match the Qwen2.5-VL-3B, B=16 streaming end-of-stream row in Table 5. The correspondence between these results and configurations needs to be established. In addition, Section 4.5 states that removing either S3 or S4 decreases the gold-answer margin by 0.75, but Figure 8(c) shows -0.75 for Q1/S3 and +0.75 for Q1/S4. Under the caption's definition, m_full - m_ablated, removing S3 should increase the margin. These issues affect reproducibility and the interpretation of slot contributions.

Questions for authors

The following are clarification requests that could change the assessment:

  1. Please specify K, the layer grouping G, the frame-level write cap, the pooled-writer cap T_mem, the steering scales, and the checkpoint correspondence for Tables 2 and 5. Why do the stated K=1 controlled results exactly match the B=16 streaming results described as using the main K=4 configuration?
  2. Do the duration-stratified gains in Figure 5 remain when Base and PREM use exactly the same sampled frame indices and buffer contents?
  3. Is the effect in Figure 8(c) defined as m_full - m_ablated? If so, please reconcile the Q1/S3 value of -0.75 with the statement that removing S3 decreases the margin, and clarify which slots support the correct answer after the correction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.