- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes KeyRec, a training-free framework for streaming and long-video understanding that separates query-agnostic visual-memory construction from query-adaptive, fixed-budget readout. It maintains a dense recent cache and a bounded historical event bank. Candidate events are constructed from visually novel tokens, consolidated using semantic similarity and temporal proximity, and evicted according to utility and temporal coverage. At query time, a text-only invocation of the frozen VLM allocates the readout budget between recent and historical evidence and determines the retrieval strategy. Experiments cover four benchmarks and three backbones, including an encoder-free model. KeyRec achieves the best or tied-best compressed performance in 13 of 15 settings with approximately 10% decoder-facing visual tokens, while maintaining a larger persistent visual memory of approximately 20%.
Critical strengths
-
A clear and practically useful separation of storage and readout. KeyRec explicitly distinguishes persistent memory capacity from the visual-token budget presented to the decoder. Its query-agnostic construction supports reuse across questions, while the recent/event split accommodates different temporal evidence requirements. This is a useful formulation for resource-constrained video understanding.
-
Broad empirical validation with informative controlled analyses. The evaluation spans streaming and offline tasks, three backbones, and both modular and encoder-free architectures. The fixed-storage event-granularity experiment, readout-allocation ablation, and full-prefix 1-FPS evaluation provide meaningful support beyond the main accuracy table. The reported latency results also make clear that compression introduces overhead at smaller inputs rather than uniformly accelerating inference.
Critical weaknesses
-
The complexity of the memory-construction pipeline is insufficiently justified, particularly for historical understanding. KeyRec combines novelty-based selection, semantic–temporal consolidation, moving-average event representations, utility-based eviction, and query-adaptive readout. Although the paper includes useful budget, granularity, serialization, and threshold analyses, these do not isolate the benefits of the core memory rules. The gains are also more consistent on realtime tasks than on historical tasks: on OVO-Bench Backward, KeyRec underperforms the strongest compressed baseline by 1.48 points on Gemma E2B and only ties it on E4B. Offline long-video gains remain positive in most settings, particularly on NEO-ov, but are generally smaller than the realtime gains. This raises the question of whether the improvements primarily come from preserving dense recent observations rather than the proposed historical event organization. Matched-budget comparisons with a simpler recent-cache-plus-history design are missing. Furthermore, threshold sensitivity is evaluated only on one backbone and subset, and the two-second merge threshold may prevent cross-frame consolidation under sparse long-video sampling. The necessity and robustness of the additional complexity therefore remain unclear.
-
The incremental benefit over closely related visual-memory frameworks remains insufficiently established. FluxMem [1], already discussed in the related work, preserves detailed recent observations and progressively compresses historical visual memory. SAVEMem [2] combines query-independent memory construction with query-adaptive retrieval. KeyRec’s event-bank updates and separation of storage and readout budgets are meaningful distinctions, but their empirical benefits over these related memory organizations are not directly established. The existing baselines are relevant, particularly CausalMem, but STC and StreamingTOM are evaluated only through their pruning/reduction components. A controlled comparison with a hierarchical visual-memory baseline would better substantiate the contribution. Although Figure 2 partially addresses storage matching, the main table matches decoder-facing tokens while allowing KeyRec a larger persistent memory.
References
[1] Yiweng Xie et al. FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding. CVPR 2026.
[2] Hang Wu et al. Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding. arXiv:2605.07897, 2026.
Questions for authors
-
Contribution beyond recent-memory preservation. Given the less consistent gains on historical tasks, how much improvement comes from the event-bank mechanisms rather than the dense recent cache? Under matched persistent-storage and readout budgets, could you compare against the same recent cache paired with uniformly retained historical tokens, and separately replace novelty-based selection, disable consolidation, or use FIFO eviction? Reporting realtime, backward-reasoning, and offline long-video results separately would clarify which components justify the added complexity.
-
Consolidation in practice and parameter robustness. Section 3.1 (“Implementation details,” p. 5) specifies 32 sampled frames per streaming prefix and 64 per complete long video. What fraction of candidates is merged, and what are the resulting event-duration distributions, under these settings and the full-prefix 1-FPS protocol in Section 3.7? In particular, how often does the two-second threshold permit cross-frame consolidation in long videos? How were the default configurations selected, and do the consolidation thresholds remain effective across tasks and sampling rates without retuning?
-
Retrieval and routing reliability. Which token representations and model layer produce the mean-pooled question embedding, and how are they aligned with visual event routes for cosine retrieval? Figure 6 shows substantial differences in routing distributions across the three backbones, particularly between Balanced and Heavy Event. Since these states change both budget allocation and retrieval strategy, do the differences reflect useful backbone-specific adaptation or routing instability? Separating these effects and comparing against a per-question oracle over the five readout states would help quantify avoidable routing losses, particularly on historical tasks.
8 comments
Rebuttal by Authors (2/4)
W2. Camparion with FluxMem and SAVEMem.
Response:
Thank you for highlighting these closely related approaches. We provide direct comparisons with FluxMem and SAVEMem, together with additional storage-matched evaluations.
(1) Advantages of KeyRec's memory design. Both FluxMem and SAVEMem maintain bounded visual memory, but differ in how historical evidence is organized and presented. FluxMem progressively compresses observations across short-, mid-, and long-term memory and decodes the retained tokens jointly. KeyRec instead maintains temporally localized events through novelty-based selection, semantic-temporal consolidation, and utility-based eviction. Crucially, it decouples persistent storage from decoder-facing readout, enabling selective access to a richer memory under a fixed per-query token budget.
SAVEMem also supports query-agnostic construction and query-adaptive retrieval, using semantic-prior-guided retention and frame-level relevance scoring. KeyRec instead organizes historical evidence into event-level units and adaptively selects relevant events or temporal coverage according to question requirements, while explicitly bounding the total decoder-facing tokens. These design choices are supported by our controlled ablations in W1/Q1: with identical recent caches, routing decisions, and budgets, replacing the event bank with uniform token-level history reduces performance. Additional ablations further support the contributions of novelty-based selection and consolidation.
(2) Empirical comparison. We compare KeyRec with FluxMem and SAVEMem on Gemma 4 E4B. KeyRec achieves the highest accuracy across all three settings, outperforming the strongest competing method across streaming video and long video understanding tasks.
| Gemma 4 E4B | OVO Backward | OVO Realtime | LongVideoBench |
|---|---|---|---|
| KeyRec | 54.45 | 64.81 | 45.77 |
| FluxMem | 51.71 | 61.51 | 45.18 |
| SAVEMem | 53.48 | 63.17 | 44.05 |
(3) Storage and readout budget fairness. In Section 3.3, we already include approximately storage-matched comparisons with the original baselines. We further evaluate FluxMem under exactly the same 480-token persistent-storage budget as KeyRec, as well as its native adaptive configuration.
As shown below, with matched storage, KeyRec uses only 224 decoder-facing tokens compared with FluxMem's 480. On E2B, KeyRec improves OVO Backward and Realtime by 2.00 and 4.64 points, respectively. On E4B, it achieves comparable Backward accuracy and improves Realtime by 3.95 points. Under FluxMem's native configuration ($\sim$32% retained tokens on average), FluxMem slightly outperforms KeyRec on E4B Backward, while KeyRec achieves higher Realtime accuracy with substantially fewer decoder-facing tokens.
| Backbone | Method | Storage | Readout | Backward | Realtime |
|---|---|---|---|---|---|
| E2B | KeyRec | 21% | 10% | 43.83 | 57.70 |
| FluxMem-480 | 21% | 21% | 41.83 | 53.06 | |
| FluxMem Native | Adaptive ( 32%) | Adaptive ( 32%) | 42.08 | 54.32 | |
| E4B | KeyRec | 21% | 10% | 54.45 | 64.81 |
| FluxMem-480 | 21% | 21% | 54.47 | 60.86 | |
| FluxMem Native | Adaptive ( 32%) | Adaptive ( 32%) | 55.23 | 62.38 |
Together, these results demonstrate that KeyRec achieves competitive historical understanding and stronger realtime performance while explicitly limiting per-query decoding cost, supporting the benefits of structured historical memory and decoupled storage/readout budgets.
Rebuttal by Authors (3/4)
Q2. Consolidation in practice and parameter robustness.
Response:
Thank you for this question. We provide additional consolidation statistics under the three evaluation protocols and extend the threshold sensitivity analysis across backbones and tasks.
(1) Consolidation behavior. The temporal threshold constrains the gap between consecutive observations rather than the total event duration. Consequently, consolidation frequency naturally depends on temporal sampling density.
-
32-frame streaming (OVO): 15% of candidates are consolidated, and 24% of questions involve at least one merge. The resulting merged events have a median temporal span of 4s. The event bank reaches capacity for 78% of questions, confirming that bounded eviction is actively exercised.
-
64-frame long-video (LVB): Under sparse sampling, almost all candidates fail the temporal proximity condition, so events predominantly retain evidence from individual sampled frames. Consolidation remains enabled but rarely activates. We also test an increased to one sampled-frame interval, merging 41% of candidates, but we observe little effect on accuracy (E4B: 45.70 vs. 45.77; E2B: 35.90 vs. 35.60). In contrast, removing the temporal constraint reduces accuracy to 43.75/33.43 on E4B/E2B, highlighting the importance of preserving temporal locality.
-
1-FPS streaming: With denser observations, 77% of candidates are consolidated, and the median per-question mean event span is 25 s. Despite the increased consolidation frequency, accuracy remains comparable with and without consolidation: 55.46/64.54 vs. 56.18/63.21 on OVO Backward/Realtime, and 68.65 vs. 68.65 on StreamingBench (E4B). Thus, the shared thresholds remain effective at the level of downstream performance across these sampling protocols.
(2) Threshold robustness configuration. In our manuscript, we use the shared defaults and to enforce semantic similarity and temporal locality, without specific tuning on benchmark, backbone, or sampling rate. We further evaluate the same and grid on additional settings. Accuracy ranges from 42.57-43.76 on E2B OVO-Backward, 57.48-57.72 on E2B OVO-Realtime, and 61.57-61.85 / 66.21-66.33 on E2B/E4B StreamingBench. The default configuration remains close to the best tested settings without task-specific tuning. Together with the results across sampling protocols, these findings support the effectiveness of a shared configuration, although alternative thresholds can yield modest improvements in individual settings.
(3) Further optimization. We acknowledge that sampling-rate-aware threshold selection is an interesting extension that could further refine memory organization. However, more aggressive consolidation does not consistently improve accuracy in our experiments, while rate-specific optimization introduces additional tuning requirements and policy complexity. Investigating this performance-complexity trade-off is beyond the scope of the present study and is left for future work.
Rebuttal by Authors (1/4)
We sincerely thank you for the constructive feedback and for recognizing the practical value of decoupling persistent memory storage from query-adaptive readout, as well as the breadth of our empirical evaluation and controlled analyses. We address your concerns point by point below.
Response:
Thank you for raising this important concern. We agree that the stronger gains on realtime tasks are consistent with the importance of preserving fine-grained recent observations. However, this does not preclude additional benefits from structured historical memory. To isolate these benefits, we conduct four controlled ablations, keeping the recent cache, routing decisions, and visual-token budget fixed:
V1: Uniform Token History, replacing the event bank with uniformly retained historical tokens;
V2: No Consolidation, disabling event merging;
V3: FIFO Eviction, replacing utility-based eviction with FIFO;
V4: Uniform Candidates, replacing novelty-based candidate selection with uniform selection.
† No consolidation occurs under sparse long-video sampling.
The results highlight two key findings: (i) Event organization contributes beyond recent-cache preservation. Compared with uniform token-level history (V1), KeyRec improves OVO Backward by 3.12 points, compared with only 0.73 points on Realtime. It also improves LongVideoBench by 2.32/1.42 points on E2B/E4B. These controlled comparisons demonstrate that the gains cannot be attributed solely to preserving recent observations. (ii) The memory-update mechanisms provide additional benefits. Disabling consolidation (V2) or novelty-based candidate selection (V4) reduces backward accuracy by 1.28 and 2.18 points, respectively. FIFO eviction (V3) remains competitive, slightly outperforming KeyRec on Realtime but underperforming on Backward and LongVideoBench. Notably, consolidation is inactive under the sparse LongVideoBench sampling protocol, explaining the identical results for KeyRec and V2. Thus, our results support complementary roles for dense recent evidence and structured historical memory, with the latter providing clearer incremental benefits for history-dependent reasoning.