acceptodds
Under review as a conference paper at ICLR 2027

Look Less, Remember More: Training-free Dynamic Encoding with Hierarchical Visual Memory for Streaming Video Understanding

Abstract

Streaming video understanding must answer questions about an evolving video stream within restricted compute budgets. However, the visual tokens grow linearly with video length, and most of them repeat the content already seen. Existing studies typically prune visual tokens in the large language model under a query-aware criterion, missing the opportunity to remove redundancy at the early encoding stage. To address this issue, we propose , a novel training-free and query-agnostic framework that relies solely on the intrinsic cross-modal semantics of a pretrained Video-LLM, which can dynamically encode the video frame, removing spatiotemporal redundancy, and effectively manage and utilize historical visual information. First, we propose a dynamic encoding approach via inter-frame temporal novelty and salience-based spatial token pruning to improve the efficiency of video representation and eliminate spatiotemporal redundancy. Second, to effectively organize and leverage historical visual information, a diversity-preserving hierarchical memory mechanism is incorporated, which can retain the most informative visual content in the memory bank without the requirement for query guidance. Extensive experiments across online and offline long-video benchmarks demonstrate that achieves superior video understanding performance with limited memory usage and lower latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, multimodal large language models, streaming inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KyS7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StreamSieve, a training-free framework for efficient streaming video understanding. It consists of two main components: (i) refined dynamic encoding for spatiotemporal redundancy, which adaptively determines whether to skip, partially encode, or fully encode incoming frames based on temporal similarity, followed by spatiotemporal saliency-guided token refinement; and (ii) query-agnostic diversity-preserving hierarchical memory, which maintains recent visual information in short-term memory and preserves diverse historical tokens in long-term memory for future query answering. Experiments on online and offline video benchmarks demonstrate improved accuracy and efficiency, with reduced visual encoding FLOPs, GPU memory usage, and inference latency.

Critical strengths
  • The most interesting module in StreamSieve is the temporal novelty-based dynamic hierarchical encoding (TNDHE), which attempts to reduce computation before visual features are fully extracted, rather than only compressing tokens after encoding. Although the proposed similarity-based dynamic encoding strategy is relatively simple, the direction is practically meaningful for streaming video understanding, where substantial redundancy exists between consecutive frames.
Critical weaknesses
  • My main concern is the limited novelty. The proposed StreamSieve is more like an engineering combination of existing techniques. In particular, FluxMem (CVPR 2026) has already explored training-free, query-agnostic spatiotemporal token reduction with hierarchical memory for streaming video understanding. The proposed Spatiotemporal Saliency-guided Token Refinement (SSTR) and Query-agnostic Diversity-preserving Hierarchical Memory (QDPHM) share similar ideas with FluxMem's token compression and hierarchical memory mechanisms. The paper should better clarify its novelty beyond FluxMem, and a direct comparison is missing.

[a] FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding, CVPR 2026.

Questions for authors
  • Clarify the main technical contributions beyond existing methods, particularly FluxMem (CVPR 2026), and explain the differences between their token refinement and hierarchical memory mechanisms.
  • The proposed SteamSieve relies on several specified hyperparameters, but their effects are largely unexplored. In particular, the three fixed thresholds used to determine the novelty of each video frame, which directly determines the encoding strategy and computational cost, yet the paper provides no justification or sensitivity analysis for these choices. Other important parameters, such as the shallow encoding depth, token retention ratios, and memory replacement threshold, are also not sufficiently studied. I would like to see whether the reported accuracy-efficiency trade-off remains stable under different settings.
  • For Similar frames, the method aggregates shallow representations from consecutive frames before propagating them through the remaining encoder layers. Could the authors clarify when the deeper layers are executed in the streaming setting? Is the accumulated representation updated and fully encoded upon each incoming frame, or only after a group of similar frames has been collected? This distinction is important for understanding the actual computational savings and online latency.
  • The efficiency analysis only compares StreamSieve with the baseline, without comparisons against existing efficient streaming video methods. Since efficiency is a major contribution of this paper, the authors should compare with relevant methods, particularly FluxMem, under comparable settings. Moreover, the evaluation should also include comparisons with simpler strategies under the same computational or visual token budget, such as uniform frame sampling, periodic full encoding, or fixed token pruning.
  • Could the authors compare the proposed method with existing approaches under the same computational or memory budget?
  • OVO-Bench includes Forward Active Responding tasks, but Table 1 only reports Real-Time Visual Perception and Backward Tracing. Could the authors clarify why the Forward Active Responding tasks are excluded from the evaluation?
  • Both backbones show some performance drops on Episodic Memory (EPM) and Action Sequence Identification (ASI) in OVO-Bench. Given that preserving historical information is one of the main goals of the proposed hierarchical memory, it would be helpful if the authors could provide some discussion or analysis of these results.
Reviewer_xwbwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StreamSieve is a training-free streaming video understanding framework combining adaptive frame encoding, visual token pruning, and hierarchical memory. Evaluations on two VLM backbones and four video benchmarks demonstrate accuracy–efficiency trade-offs through reduced encoding computation and context size.

Critical strengths
  • Early-stage computational savings. The framework reduces redundant computation during visual encoding, rather than only compressing tokens afterward. The reported encoder FLOPs and latency reductions make this a practically useful contribution for streaming video processing.
Critical weaknesses
  1. Limited methodological novelty. The framework largely combines heuristic frame routing, token scoring, and memory replacement. The paper does not sufficiently establish a distinct methodological contribution beyond their integration.
  2. Insufficient controlled comparisons. Memory-only ablations and comparisons against simpler sampling and memory policies under matched budgets are missing. Consequently, the necessity of the proposed components remains unclear.
  3. Weak evidence for improved historical memory. Both backbones show lower Episodic Memory and Action Sequence Identification scores than their baselines. The backward-tracing gains primarily come from Hallucination Detection, providing limited support for improved retention of historical evidence.
  4. Reproducibility and backbone ambiguity. The feature accumulation procedure and key memory hyperparameters are insufficiently specified. Furthermore, the reproducibility statement mentions a streaming-tuned checkpoint, conflicting with the claim that the backbone was not trained for streaming and raising concerns about baseline comparability.
  5. Incomplete efficiency evaluation. Query-time TTFT does not capture preceding encoding and memory maintenance costs. Without streaming throughput and matched-budget accuracy–latency comparisons, the practical advantage over simpler alternatives remains uncertain.
Questions for authors
  1. Backbone clarification: Which exact checkpoints were used, and were the baselines evaluated using the same checkpoints?
  2. Encoding clarification: When are accumulated shallow features passed through the remaining encoder layers, particularly if a query arrives mid-accumulation?
  3. Controlled comparisons: How does the method compare with memory-only and simpler sampling/memory policies under matched budgets?
  4. Memory effectiveness: How do you reconcile the historical-memory claims with reduced Episodic Memory and Action Sequence Identification scores?
  5. Efficiency clarification: What is the streaming throughput when encoding, memory updates, and retrieval overhead are included?
Reviewer_KvijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes StreamSieve, a training-free framework for streaming video understanding. It reduces redundant visual computation through dynamic encoding and token pruning, while using hierarchical short- and long-term memory to preserve useful historical information. Experiments show competitive accuracy with lower memory usage and latency.

Critical strengths

The paper studies a practical problem in streaming video understanding, considering visual encoding cost, limited memory capacity, and unknown future queries. The proposed framework combines adaptive encoding, token filtering, and short-/long-term memory into a coherent training-free pipeline.

Critical weaknesses
  1. Table 4 shows that memory improves performance, but it is unclear whether the gain comes from generic memory management or the proposed long-term selection and replacement strategy.

  2. The comparison protocol needs clarification. Different backbones, sampling strategies, and resource budgets may affect both accuracy and efficiency. Results under a unified constraint, such as the same visual-token budget or GPU memory limit, would make the comparison more convincing.

  3. The motivation for the long-term memory novelty metric is not fully justified. A larger cosine distance does not necessarily imply higher semantic value or usefulness for future questions.

Questions for authors
  1. EarlyTom has already explored frame compression within the visual encoder followed by spatial token filtering. Could the authors clarify how the proposed hierarchical encoding mechanism differs from EarlyTom and what additional problems these differences address?

  2. Section 4.1 and the main result tables refer to Qwen2.5-VL-7B, while the reproducibility statement on page 10 mentions the released TimeChat-Online-7B checkpoint. Is this a typo or an inconsistency in the experimental setup? Please clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.