acceptodds
Under review as a conference paper at ICLR 2027

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Abstract

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose **KeyRec**, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add–merge–evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder–projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21–18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Streaming Video Understanding, Visual Token Compression, Visual Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dZJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KeyRec, a training-free framework for streaming and long-video understanding that separates query-agnostic visual-memory construction from query-adaptive, fixed-budget readout. It maintains a dense recent cache and a bounded historical event bank. Candidate events are constructed from visually novel tokens, consolidated using semantic similarity and temporal proximity, and evicted according to utility and temporal coverage. At query time, a text-only invocation of the frozen VLM allocates the readout budget between recent and historical evidence and determines the retrieval strategy. Experiments cover four benchmarks and three backbones, including an encoder-free model. KeyRec achieves the best or tied-best compressed performance in 13 of 15 settings with approximately 10% decoder-facing visual tokens, while maintaining a larger persistent visual memory of approximately 20%.

Critical strengths
  1. A clear and practically useful separation of storage and readout. KeyRec explicitly distinguishes persistent memory capacity from the visual-token budget presented to the decoder. Its query-agnostic construction supports reuse across questions, while the recent/event split accommodates different temporal evidence requirements. This is a useful formulation for resource-constrained video understanding.

  2. Broad empirical validation with informative controlled analyses. The evaluation spans streaming and offline tasks, three backbones, and both modular and encoder-free architectures. The fixed-storage event-granularity experiment, readout-allocation ablation, and full-prefix 1-FPS evaluation provide meaningful support beyond the main accuracy table. The reported latency results also make clear that compression introduces overhead at smaller inputs rather than uniformly accelerating inference.

Critical weaknesses
  1. The complexity of the memory-construction pipeline is insufficiently justified, particularly for historical understanding. KeyRec combines novelty-based selection, semantic–temporal consolidation, moving-average event representations, utility-based eviction, and query-adaptive readout. Although the paper includes useful budget, granularity, serialization, and threshold analyses, these do not isolate the benefits of the core memory rules. The gains are also more consistent on realtime tasks than on historical tasks: on OVO-Bench Backward, KeyRec underperforms the strongest compressed baseline by 1.48 points on Gemma E2B and only ties it on E4B. Offline long-video gains remain positive in most settings, particularly on NEO-ov, but are generally smaller than the realtime gains. This raises the question of whether the improvements primarily come from preserving dense recent observations rather than the proposed historical event organization. Matched-budget comparisons with a simpler recent-cache-plus-history design are missing. Furthermore, threshold sensitivity is evaluated only on one backbone and subset, and the two-second merge threshold may prevent cross-frame consolidation under sparse long-video sampling. The necessity and robustness of the additional complexity therefore remain unclear.

  2. The incremental benefit over closely related visual-memory frameworks remains insufficiently established. FluxMem [1], already discussed in the related work, preserves detailed recent observations and progressively compresses historical visual memory. SAVEMem [2] combines query-independent memory construction with query-adaptive retrieval. KeyRec’s event-bank updates and separation of storage and readout budgets are meaningful distinctions, but their empirical benefits over these related memory organizations are not directly established. The existing baselines are relevant, particularly CausalMem, but STC and StreamingTOM are evaluated only through their pruning/reduction components. A controlled comparison with a hierarchical visual-memory baseline would better substantiate the contribution. Although Figure 2 partially addresses storage matching, the main table matches decoder-facing tokens while allowing KeyRec a larger persistent memory.

References

[1] Yiweng Xie et al. FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding. CVPR 2026.

[2] Hang Wu et al. Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding. arXiv:2605.07897, 2026.

Questions for authors
  1. Contribution beyond recent-memory preservation. Given the less consistent gains on historical tasks, how much improvement comes from the event-bank mechanisms rather than the dense recent cache? Under matched persistent-storage and readout budgets, could you compare against the same recent cache paired with uniformly retained historical tokens, and separately replace novelty-based selection, disable consolidation, or use FIFO eviction? Reporting realtime, backward-reasoning, and offline long-video results separately would clarify which components justify the added complexity.

  2. Consolidation in practice and parameter robustness. Section 3.1 (“Implementation details,” p. 5) specifies 32 sampled frames per streaming prefix and 64 per complete long video. What fraction of candidates is merged, and what are the resulting event-duration distributions, under these settings and the full-prefix 1-FPS protocol in Section 3.7? In particular, how often does the two-second threshold permit cross-frame consolidation in long videos? How were the default configurations selected, and do the consolidation thresholds remain effective across tasks and sampling rates without retuning?

  3. Retrieval and routing reliability. Which token representations and model layer produce the mean-pooled question embedding, and how are they aligned with visual event routes for cosine retrieval? Figure 6 shows substantial differences in routing distributions across the three backbones, particularly between Balanced and Heavy Event. Since these states change both budget allocation and retrieval strategy, do the differences reflect useful backbone-specific adaptation or routing instability? Separating these effects and comparing against a per-question oracle over the five readout states would help quantify avoidable routing losses, particularly on historical tasks.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We sincerely thank you for the constructive feedback and for recognizing the practical value of decoupling persistent memory storage from query-adaptive readout, as well as the breadth of our empirical evaluation and controlled analyses. We address your concerns point by point below.

W1/Q1. Contribution of Historical Event Memory

Response:

Thank you for raising this important concern. We agree that the stronger gains on realtime tasks are consistent with the importance of preserving fine-grained recent observations. However, this does not preclude additional benefits from structured historical memory. To isolate these benefits, we conduct four controlled ablations, keeping the recent cache, routing decisions, and visual-token budget fixed:

  • V1: Uniform Token History, replacing the event bank with uniformly retained historical tokens;

  • V2: No Consolidation, disabling event merging;

  • V3: FIFO Eviction, replacing utility-based eviction with FIFO;

  • V4: Uniform Candidates, replacing novelty-based candidate selection with uniform selection.

Benchmark (VLM)KeyRecV1V2V3V4
OVO Backward (Gemma 4 E4B)54.4551.3353.1753.6852.27
OVO Realtime (Gemma 4 E4B)64.8164.0863.7165.0063.84
LongVideoBench (Gemma 4 E4B)45.7744.3545.77 †^\dagger44.2845.62
LongVideoBench (Gemma 4 E2B)35.6033.2835.60 †^\dagger34.0334.26

†^\dagger No consolidation occurs under sparse long-video sampling.

The results highlight two key findings: (i) Event organization contributes beyond recent-cache preservation. Compared with uniform token-level history (V1), KeyRec improves OVO Backward by 3.12 points, compared with only 0.73 points on Realtime. It also improves LongVideoBench by 2.32/1.42 points on E2B/E4B. These controlled comparisons demonstrate that the gains cannot be attributed solely to preserving recent observations. (ii) The memory-update mechanisms provide additional benefits. Disabling consolidation (V2) or novelty-based candidate selection (V4) reduces backward accuracy by 1.28 and 2.18 points, respectively. FIFO eviction (V3) remains competitive, slightly outperforming KeyRec on Realtime but underperforming on Backward and LongVideoBench. Notably, consolidation is inactive under the sparse LongVideoBench sampling protocol, explaining the identical results for KeyRec and V2. Thus, our results support complementary roles for dense recent evidence and structured historical memory, with the latter providing clearer incremental benefits for history-dependent reasoning.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

W2. Camparion with FluxMem and SAVEMem.

Response:

Thank you for highlighting these closely related approaches. We provide direct comparisons with FluxMem and SAVEMem, together with additional storage-matched evaluations.

(1) Advantages of KeyRec's memory design. Both FluxMem and SAVEMem maintain bounded visual memory, but differ in how historical evidence is organized and presented. FluxMem progressively compresses observations across short-, mid-, and long-term memory and decodes the retained tokens jointly. KeyRec instead maintains temporally localized events through novelty-based selection, semantic-temporal consolidation, and utility-based eviction. Crucially, it decouples persistent storage from decoder-facing readout, enabling selective access to a richer memory under a fixed per-query token budget.

SAVEMem also supports query-agnostic construction and query-adaptive retrieval, using semantic-prior-guided retention and frame-level relevance scoring. KeyRec instead organizes historical evidence into event-level units and adaptively selects relevant events or temporal coverage according to question requirements, while explicitly bounding the total decoder-facing tokens. These design choices are supported by our controlled ablations in W1/Q1: with identical recent caches, routing decisions, and budgets, replacing the event bank with uniform token-level history reduces performance. Additional ablations further support the contributions of novelty-based selection and consolidation.

(2) Empirical comparison. We compare KeyRec with FluxMem and SAVEMem on Gemma 4 E4B. KeyRec achieves the highest accuracy across all three settings, outperforming the strongest competing method across streaming video and long video understanding tasks.

Gemma 4 E4BOVO BackwardOVO RealtimeLongVideoBench
KeyRec54.4564.8145.77
FluxMem51.7161.5145.18
SAVEMem53.4863.1744.05

(3) Storage and readout budget fairness. In Section 3.3, we already include approximately storage-matched comparisons with the original baselines. We further evaluate FluxMem under exactly the same 480-token persistent-storage budget as KeyRec, as well as its native adaptive configuration.

As shown below, with matched storage, KeyRec uses only 224 decoder-facing tokens compared with FluxMem's 480. On E2B, KeyRec improves OVO Backward and Realtime by 2.00 and 4.64 points, respectively. On E4B, it achieves comparable Backward accuracy and improves Realtime by 3.95 points. Under FluxMem's native configuration ($\sim$32% retained tokens on average), FluxMem slightly outperforms KeyRec on E4B Backward, while KeyRec achieves higher Realtime accuracy with substantially fewer decoder-facing tokens.

BackboneMethodStorageReadoutBackwardRealtime
E2BKeyRec∼\sim 21%10%43.8357.70
FluxMem-480∼\sim 21%∼\sim 21%41.8353.06
FluxMem NativeAdaptive (∼\sim 32%)Adaptive (∼\sim 32%)42.0854.32
E4BKeyRec∼\sim 21%10%54.4564.81
FluxMem-480∼\sim 21%∼\sim 21%54.4760.86
FluxMem NativeAdaptive (∼\sim 32%)Adaptive (∼\sim 32%)55.2362.38

Together, these results demonstrate that KeyRec achieves competitive historical understanding and stronger realtime performance while explicitly limiting per-query decoding cost, supporting the benefits of structured historical memory and decoupled storage/readout budgets.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

Q2. Consolidation in practice and parameter robustness.

Response:

Thank you for this question. We provide additional consolidation statistics under the three evaluation protocols and extend the threshold sensitivity analysis across backbones and tasks.

(1) Consolidation behavior. The temporal threshold δ=2\delta=2 constrains the gap between consecutive observations rather than the total event duration. Consequently, consolidation frequency naturally depends on temporal sampling density.

  • 32-frame streaming (OVO): 15% of candidates are consolidated, and 24% of questions involve at least one merge. The resulting merged events have a median temporal span of 4s. The event bank reaches capacity for 78% of questions, confirming that bounded eviction is actively exercised.

  • 64-frame long-video (LVB): Under sparse sampling, almost all candidates fail the temporal proximity condition, so events predominantly retain evidence from individual sampled frames. Consolidation remains enabled but rarely activates. We also test an increased δ\delta to one sampled-frame interval, merging ∼\sim 41% of candidates, but we observe little effect on accuracy (E4B: 45.70 vs. 45.77; E2B: 35.90 vs. 35.60). In contrast, removing the temporal constraint reduces accuracy to 43.75/33.43 on E4B/E2B, highlighting the importance of preserving temporal locality.

  • 1-FPS streaming: With denser observations, 77% of candidates are consolidated, and the median per-question mean event span is 25 s. Despite the increased consolidation frequency, accuracy remains comparable with and without consolidation: 55.46/64.54 vs. 56.18/63.21 on OVO Backward/Realtime, and 68.65 vs. 68.65 on StreamingBench (E4B). Thus, the shared thresholds remain effective at the level of downstream performance across these sampling protocols.

(2) Threshold robustness configuration. In our manuscript, we use the shared defaults γ=0.75\gamma=0.75 and δ=2\delta=2 to enforce semantic similarity and temporal locality, without specific tuning on benchmark, backbone, or sampling rate. We further evaluate the same γ∈{0.65,0.75,0.85}\gamma\in\{0.65,0.75,0.85\} and δ∈{1,2,4}\delta\in\{1,2,4\} grid on additional settings. Accuracy ranges from 42.57-43.76 on E2B OVO-Backward, 57.48-57.72 on E2B OVO-Realtime, and 61.57-61.85 / 66.21-66.33 on E2B/E4B StreamingBench. The default configuration remains close to the best tested settings without task-specific tuning. Together with the results across sampling protocols, these findings support the effectiveness of a shared configuration, although alternative thresholds can yield modest improvements in individual settings.

(3) Further optimization. We acknowledge that sampling-rate-aware threshold selection is an interesting extension that could further refine memory organization. However, more aggressive consolidation does not consistently improve accuracy in our experiments, while rate-specific optimization introduces additional tuning requirements and policy complexity. Investigating this performance-complexity trade-off is beyond the scope of the present study and is left for future work.

Reviewer_vqFmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KeyRec, a training-free framework for streaming and long-video understanding with bounded visual memory. It keeps fine-grained recent observations in a cache and historical evidence in an event bank. The bank is updated through novelty-based candidate selection, insertion, consolidation, and eviction. At inference time, a text-only router allocates a fixed visual-token budget between the two memories based on the query. The evaluation covers four video-understanding benchmarks and three VLM backbones, including a native one-vision architecture. With a decoder-facing visual-token budget of approximately 10% of the dense input, KeyRec achieves the best or tied-best performance among compressed methods in 13 of 15 settings.

Critical strengths
  1. Strong results with a limited readout budget. KeyRec performs well across streaming and long-video benchmarks while substantially reducing the visual tokens passed to the decoder. The results cover multiple VLM backbones, and the fixed-budget streaming experiments provide further evidence of its effectiveness.
  2. No additional training and applicability across architectures. KeyRec uses the same memory construction and readout approach across different backbones without additional training. The results on the native one-vision NEO-ov architecture are particularly interesting, since many visual-token compression methods rely on architecture-specific vision encoder features or attention signals.
Critical weaknesses
  1. The contribution beyond closely related work, particularly StreamForest, is unclear. StreamForest (NeurIPS 2025) already uses a Persistent Event Memory Forest for historical information and a Fine-grained Spatiotemporal Window for recent observations. It also considers semantic similarity and temporal distance when organizing and consolidating historical evidence. This overlaps with two central parts of KeyRec: separating recent and historical memory, and consolidating events using semantic and temporal information. KeyRec does differ in its training-free formulation, model-facing visual-token representations, bounded event bank, and query-adaptive readout under a fixed budget. However, the paper does not clearly explain which of these differences constitute its main methodological contribution. I would like to see a direct technical comparison with StreamForest that explains what these changes enable and provides evidence of their benefits.
  2. The ablations do not establish why the proposed event-memory mechanisms are needed. The full framework performs well, but the experiments leave open how much of the improvement comes from event organization. Simpler alternatives are missing, including novelty-based token selection without event grouping, semantic-only consolidation, and alternative eviction policies. Without these comparisons, the gains could come from the recent cache, informative-token selection, or query-adaptive readout. The ablations on event granularity, consolidation thresholds, and memory serialization help explain how to configure KeyRec. They do less to test whether its core design choices are necessary. I also miss a simple recency-only streaming baseline, such as SimpleStream, using the same backbone and resource budgets. Figure 4 includes a recent-only readout ablation, but this does not fully answer whether maintaining the event bank is better than using a standalone sliding window. The memory budgets complicate this comparison. KeyRec stores approximately 20% of the dense visual tokens in persistent memory and passes 10% to the decoder, whereas most baselines retain approximately 10%. The budget-scaling comparison is useful, but matching both persistent storage and decoder-facing budgets would help determine whether the gains come from event organization or from having more candidate tokens available.
Questions for authors

Q1. What are the main methodological differences between KeyRec and StreamForest? Both separate recent and historical memory and use semantic and temporal information to consolidate events. Which parts of KeyRec add capabilities beyond these shared design choices, and what evidence shows that those differences matter in practice?

Q2. Can the authors test the event bank against simpler memory policies? For example, could the event bank be replaced with a token-level memory while keeping the recent cache, router, and visual-token budgets fixed? Separate ablations of novelty-based selection, semantic-temporal consolidation, and eviction would also help show which components account for the gains.

Q3. What causes the failures on counting and repeated-event reasoning? On the Counting (Cnt) category of StreamingBench with Gemma E2B (Table 8), KeyRec scores 17.62, compared with 43.01 for Vanilla and 31.09 for CausalMem. This is a large drop and deserves a closer look. Could novelty-based selection and event consolidation be discarding repetitive observations that are needed for counting? Could uniform subsampling within merged events be losing fine-grained evidence? Representative failure cases would help, along with an analysis of whether the relevant evidence is lost during memory construction or missed during query-time retrieval.

Optional: Uncertainty estimates would help interpret small performance differences, though this is not a main concern.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We sincerely thank you for the constructive feedback and for recognizing KeyRec's strong performance and its training-free applicability across different VLM architectures. We address your concerns regarding methodological novelty, event-memory design, budget fairness, and counting failures point by point below.

W1/Q1. Differences between KeyRec and StreamForest

Response:

Thank you for highlighting this closely related work. While StreamForest and KeyRec both distinguish recent and historical evidence, we would like to note several important differences.

  • Training-free deployment and model compatibility. As you note, KeyRec is training-free. We would like to further emphasize the practical advantages of this distinction. While StreamForest fine-tunes its projector and LLM for streaming video understanding, KeyRec can be directly integrated into existing pretrained VLMs without additional training data or parameter updates. This enables the same memory construction and readout mechanism to be applied across different backbones without model-specific fine-tuning.

  • Event construction and maintenance. A key advantage of KeyRec is its compatibility with native one-vision architectures. StreamForest's ToMe-based merging produces fused visual features that do not directly preserve the original spatiotemporal positions required by NEO-ov, making its original memory mechanism difficult to transfer without additional architectural adaptations. In contrast, KeyRec selects and retains original model-facing visual tokens together with their positional metadata, enabling direct integration with NEO-ov through lightweight positional bookkeeping. Additionally, KeyRec's novelty-driven selection, semantic-temporal consolidation, and utility-based eviction aim to preserve distinctive, temporally localized evidence within a bounded event bank, without repeatedly merging visual features.

  • Decoupled storage and query-adaptive readout. A key distinction is that StreamForest feeds the same retained features to the LLM for different queries at a given time. KeyRec instead decouples query-agnostic memory construction from query-adaptive readout: its text-only router dynamically allocates a fixed decoder-facing budget between recent and historical evidence, retrieving events by query relevance or temporal coverage. This allows the same persistent memory to provide different evidence for different questions, without reprocessing historical frames.

We further evaluate a training-free version of StreamForest and additionally replace KeyRec's historical event bank with StreamForest's PEMF memory while retaining its recent cache and query-adaptive readout, denoted as KeyRec w. PEMF. Under the same readout budget, KeyRec outperforms StreamForest in five of six settings. The PEMF replacement also achieves competitive results, suggesting that KeyRec's memory and readout design can accommodate alternative event representations. Combined with the query-adaptive allocation ablation (Figure 4) and successful adaptation to NEO-ov, these results highlight the benefits of decoupled readout, distinctive event organization, and compatibility with different VLM architectures.

BackboneMethodOVO BackwardOVO RealtimeStreamingBench
E2BStreamForest43.2853.6360.09
KeyRec w. PEMF44.1057.5861.41
KeyRec43.8357.7061.29
E4BStreamForest52.8361.4166.61
KeyRec w. PEMF52.2963.2566.73
KeyRec54.4564.8166.49
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2/Q2. Additional ablations & SimpleStream baseline & Budget-scaling comparison.

Response:

Thank you for raising these important points. We address them one by one below.

(1) Ablation variants. We evaluate six variants while keeping the recent cache, routing decisions, and visual-token budgets fixed:

  • V1: uniform token-level history without event grouping;

  • V2: novelty-selected token-level history without event grouping;

  • V3: no event consolidation;

  • V4: semantic-only consolidation without temporal constraints;

  • V5: FIFO eviction instead of utility-based eviction;

  • V6: uniform candidate selection instead of novelty-based selection.

As shown below, the full KeyRec achieves the best OVO-Backward and overall average across all variants. The results highlight two key findings: (i) Event organization is beneficial for historical reasoning. Replacing the event bank with token-level history (V1/V2) reduces backward accuracy by 3.12/1.80 points. This indicates that novelty-based token selection alone does not recover the benefits of structured event memory. (ii) Individual memory mechanisms provide further improvements. Removing consolidation (V3), temporal constraints (V4), or novelty-based candidate selection (V6) further reduces backward accuracy by 1.28, 2.66, and 2.18 points, respectively. FIFO eviction (V5) remains competitive, but yields a lower overall average. Notably, the variants remain strong on OVO Realtime, suggesting that the recent cache and query-adaptive readout already provide effective real-time understanding, while structured event memory offers additional benefits, particularly for history-dependent reasoning. Together, these results demonstrate the complementary roles of KeyRec's memory construction and readout mechanisms.

Gemma 4 E4BKeyRecV1V2V3V4V5V6
OVO Backward54.4551.3352.6553.1751.7953.6852.27
OVO Realtime64.8164.0864.2063.7164.2365.0063.84
OVO Avg.59.6357.7158.4358.4458.0159.3458.06

(2) Comparison with SimpleStream. We would like to note that though simple and effective, SimpleStream is specifically designed for streaming video understanding and relies on a sliding window of densely sampled recent frames (e.g., 1 FPS). As shown below, it slightly outperforms KeyRec on StreamingBench, benefiting from its dense sampling of the most recent frames, whereas KeyRec samples frames across the video prefix. KeyRec with the same densely SimpleStream-sampled recent observations can outperform SimpleStream. This suggests that SimpleStream's advantage is largely associated with its recent-frame sampling protocol, rather than an inherent advantage of its memory policy.

However, its dense recent sampling strategy cannot preserve long-range historical evidence. On LongVideoBench, KeyRec substantially outperforms SimpleStream by +7.40 on E2B and +7.25 on E4B. These results highlight KeyRec's broader applicability to both streaming and long-video understanding, where maintaining historical evidence is essential.

BenchmarkMethodGemma 4 E2BGemma 4 E4B
StreamingBenchSimpleStream62.9367.45
KeyRec61.2966.49
KeyRec with SimpleStream's recent cache64.0568.73
LongVideoBenchSimpleStream28.2038.52
KeyRec35.6045.77

(3) Budget-controlled comparison. We agree that distinguishing persistent storage from decoder-facing readout is important for a fair comparison. Our main experiments match the decoder-facing budget (10%), while KeyRec maintains a larger persistent memory for query-adaptive retrieval. Importantly, we have already investigated this concern through approximately matched-storage comparisons in Section 3.3: KeyRec at 5% readout is compared with baselines at 10%, and KeyRec at 10% with baselines at 20%. In both settings, KeyRec achieves higher accuracy despite using only half as many decoder-facing visual tokens under comparable persistent-storage budgets. This shows that its gains cannot be attributed simply to storing more visual tokens. Furthermore, our additional component ablations keep both storage and readout budgets fixed, allowing us to examine the contribution of event-memory organization separately.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q3. Counting Failures and Causes.

Response:

Thank you for highlighting this interesting failure case. We would like to provide additional analysis showing that this large gap on Gemma 4 E2B is primarily associated with query-time readout rather than event-memory construction.

Specifically, the E2B router assigns 92% of Counting questions to the Balanced state, which uses relevance-based event retrieval, whereas E4B assigns 91% to the Heavy Event state, which uses temporal-coverage retrieval. Counting repeated occurrences requires broad temporal coverage, which relevance-based selection may fail to provide.

To investigate this, we evaluated all five readout policies using the same stored memory in a separate controlled diagnostic run. Switching to temporal-coverage readout (Heavy Event/All Event) improves E2B Counting (Cnt) accuracy by + 10.4 points. This within-run improvement suggests that a substantial part of the degradation is attributable to the selected readout policy, rather than irreversible evidence loss during memory construction. Importantly, this routing difference is consistent with the cross-backbone results: KeyRec outperforms CausalMem on Counting with both E4B and NEO-ov 2B (Table 8). These findings suggest a model- and task-dependent routing limitation rather than a systematic weakness of KeyRec's event memory relative to other compressed methods. Thank you again for this insightful observation and we will incorporate the analysis and corresponding explanations into the revised manuscript.

Optional. Uncertainty estimates.

Response:

Thank you for this suggestion. Our evaluation follows standard benchmark protocols, and we agree that uncertainty estimates would provide additional context for interpreting small performance differences. We will consider including such analysis in future evaluations.

Reviewer_Y9QQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KeyRec, a historical memory management method for streaming VLMs. It partitions the memory space into recent frames and historical key events, and then determines how much information to retrieve from each part according to the query. As an effective historical memory management approach, this method can alleviate the limitation of constrained historical memory space in long-video understanding.

Critical strengths
  1. The experiments in this paper are comprehensive and cover a wide range. It achieves optimal compression performance under various different settings and is validated on streaming benchmarks, which effectively supports the superiority of the proposed method.
  2. This paper reveals scenarios where existing streaming VLM methods may lead to significant performance degradation, which has certain practical significance.
Critical weaknesses
  1. This paper proposes a training-free method. It needs to be validated on multiple mainstream streaming VLMs and various benchmarks to verify its generalization and demonstrate the advantages of the proposed method. However, limitations still remain at present.
  2. The compression of the method proposed in this paper is still lossy compression. In long-video benchmark tests, KeyRec consistently underperforms the dense Vanilla model. Although the performance trade-off for efficiency is acceptable, it remains a limitation of the proposed method.
Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank you for the positive assessment of our work, particularly for recognizing the comprehensive experimental evaluation and the practical significance of our findings. We address the concerns regarding generalization and lossy compression point by point below.

W1. This paper proposes a training-free method. It needs to be validated on multiple mainstream streaming VLMs and various benchmarks to verify its generalization and demonstrate the advantages of the proposed method. However, limitations still remain at present.

Response:

Thank you for raising this concern. We would like to note that KeyRec is a training-free visual memory framework applicable to both streaming and long-video understanding, rather than a dedicated streaming VLM. Specialized streaming VLMs often involve streaming-specific training and model-dependent memory or inference mechanisms [1,2], whereas KeyRec operates on frozen VLM backbones without additional training. Our original experiments already cover three backbones across two distinct architectures and four benchmarks, including fixed-budget full-prefix streaming evaluation. To further validate KeyRec's generalization, we additionally evaluate on Qwen3-VL-4B. The results provide further evidence of cross-backbone generalization, while integration with specialized streaming VLM systems remains an interesting direction for future work.

[1] Chen et.al. VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024

[2] Zeng et al. Streamforest: Efficient online video understanding with persistent event memory. NeurIPS, 2026.

MethodOVO BackwardOVO RealtimeStreamingBenchLongVideoBenchVideo-MME-v2
Vanilla39.4261.1468.0159.2426.09
STC-Pruner38.8255.7161.2151.0122.22
StreamingTOM-CTR37.9250.2959.2949.6621.00
CausalMem40.2665.9567.4152.0322.38
KeyRec47.8070.7373.9849.9622.72

W2. The compression of the method proposed in this paper is still lossy compression. In long-video benchmark tests, KeyRec consistently underperforms the dense Vanilla model. Although the performance trade-off for efficiency is acceptable, it remains a limitation of the proposed method.

Response:

Thank you for bringing up this point. We acknowledge that visual-token compression can discard information and lead to an accuracy gap relative to dense Vanilla, particularly on long-video benchmarks. However, we do not claim to achieve lossless compression or consistently outperform uncompressed inference. Instead, our goal is to improve the accuracy-efficiency trade-off under bounded visual-memory and decoder-facing token budgets. As shown in Table 1, KeyRec generally outperforms competing efficient methods at a matched 10% decoder-facing token budget, achieving the best compressed performance in five of six original long-video settings. Figure 3 further demonstrates its favorable scaling as the number of sampled frames increases, while Table 2 shows that KeyRec remains feasible at 512 frames, where dense Vanilla runs out of memory. These results support the effectiveness of structured visual memory in preserving useful evidence under constrained budgets, while acknowledging the inherent limitations of lossy compression.

AuthorsOpenReview ↗

A kind reminder

Dear Reviewer Y9QQ,

We wanted to briefly follow up before the discussion period ends. We hope that our rebuttal has helped clarify the concerns raised in your review. Please let us know if any questions remain or if further evidence or explanation would be useful.

Thank you again for your time and consideration.

Best,

Authors

AuthorsOpenReview ↗

A kind reminder

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its end. We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate any further feedback you may have before the discussion closes. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.