- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PReM, a framework for augmenting frozen vision-language models with recurrent memory for long-video understanding. During query-independent video ingestion, a gated associative update compresses frame features into a fixed-size set of memory matrices. At question-answering time, question-conditioned memory reads steer the keys and values of existing non-visual prefix tokens during prefill, without introducing additional memory tokens.
Critical strengths
- The memory-to-decoder interface is a useful and clearly motivated design. PReM separates query-independent video ingestion from question-conditioned memory access. Steering existing prefix K/V representations provides a practical way to expose recurrent memory to a frozen decoder without expanding the prompt with memory tokens.
- The paper makes a meaningful effort to isolate the proposed interface. The controlled comparison includes a Token-Readout baseline that uses the same recurrent writer, ingested stream, and trainable parameter budget.
- Several practical details are reported transparently. The paper documents the training subset, identifier-based video decontamination, state size, trainable parameter counts, and separate ingestion and answering costs.
Critical weaknesses
- The configuration associated with a central result is inconsistent across the manuscript. Appendix A.4 explicitly states that both Token-Readout and PReM use in Table 2. However, all six PReM benchmark scores in Table 2 exactly match the streaming results in Table 5, while Appendix A.6 states that the main checkpoints use . These settings also imply different recurrent-state capacities. The manuscript does not establish whether the two tables reuse the same checkpoint, whether one configuration description is incorrect, or whether distinct models independently produced the same rounded results. This ambiguity affects the interpretation of both the controlled comparison and the main results and should be resolved with an explicit checkpoint-to-configuration mapping.
- The experiments establish performance on sparsely sampled long videos more clearly than retention over long recurrent streams. Evaluation uniformly subsamples long videos to at most 240 frames and asks questions only after ingestion finishes. Thus, a video lasting over an hour does not necessarily produce more recurrent updates than a much shorter video.
- The efficiency advantage needs to be qualified by workload and latency. The small additional GPU memory footprint is well supported, but the measured runtime overhead is appreciable.
Questions for authors
- Can Figure 5 be repeated with exactly the same decoder frame indices for Base and PReM? Do the larger gains on longer videos persist?
- How does accuracy change when the recurrent stream substantially exceeds 240 frames? Can the authors evaluate retention as a function of evidence age and intervening distractor frames?
- How does the proposed writer compare with temporal pooling, an exponential moving average, or a simpler associative update when all use the same prefix-steering readout? How sensitive is Token-Readout to token count and insertion position?