acceptodds
Under review as a conference paper at ICLR 2027

EviStream: Evidence-Preserving Memory for Streaming Video Understanding

Abstract

Streaming video assistants must continuously process an unbounded visual stream before knowing what users may ask in the future, requiring efficient memory without losing evidence needed for later reasoning. Existing approaches mainly reduce visual computation or build retrievable memories, but often leave unclear which visual evidence should survive compression and how it should be recovered when needed. We introduce EviStream, a training-free evidence-preserving memory framework based on MLLMs. EviStream follows the principle of preserving residual information rather than repeated observations through complementary semantic memory and visual evidence. Incremental Semantic Memory (ISM) organizes the incoming stream into adaptive windows, removes temporal redundancy, and incrementally consolidates each window into semantic deltas, yielding a compact evolving textual history. To complement this semantic abstraction, Memory-Aligned Evidence Keyframe Selection (MEKS) preserves sparse visual evidence according to temporal novelty, the information gap left by semantic consolidation, and query relevance when available. Once a query arrives, Continual Evidence-Polling Reasoning (CEPR) retrieves relevant semantic memory and complementary visual evidence together with live observations, while continuously consolidating and re-retrieving new observations when current evidence is insufficient. EviStream achieves state-of-the-art real-time understanding performance on StreamingBench and OVO-Bench, reaching 84.1% and 76.8% accuracy, respectively, without additional training. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, VLLM, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cChpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EviStream, a training-free framework for streaming video understanding. ISM incrementally records textual summaries, while MEKS selects keyframes based on temporal novelty, text-gap feedback, and query relevance. After a query arrives, the system retrieves text and associated frames, continues updating memory, and repeatedly decides whether to answer or wait.

Critical strengths

S1: The problem is important, and the framework requires no additional training.

Reducing the token footprint of continuously arriving video while preserving evidence for future questions is an important and practically relevant problem. Organizing complementary textual summaries and keyframes chronologically can reduce redundant visual tokens while retaining event context and visual details. The framework requires no additional training or parameter updates.

Critical weaknesses

W1: The advantages over existing methods are unclear.

The closely related Nar-KFC first selects keyframes and then uses text to supplement the remaining content, whereas this paper first generates textual summaries and then selects keyframes. Does this change in processing order provide a substantive advantage? Moreover, methods such as TimeChat-Online compress similar visual content and retain differences. This is conceptually similar to the proposed “abstraction plus supplementation”, with the main difference being that this work summarizes shared content as text. More concerningly, the experimental comparisons used to demonstrate this design’s advantage are not sufficiently fair: this work uses the stronger Qwen3.5-9B without comparing these methods under the comparable model. Therefore, it is difficult to determine whether the gains come from the proposed memory mechanism.

W2: The retrieval logic at answer time is confusing.

The method uses keyframes to preserve visual information omitted from textual summaries, yet retrieves text first and then accesses keyframes from the corresponding windows. If a summary omits the object or event asked about, the relevant window may not be retrieved. As a result, even if the visual evidence needed to answer the question has been preserved, it may remain inaccessible to the answering model.

Questions for authors

Q1: Can the comparison with Nar-KFC use matched input budgets?

Table 3 lists Nar-KFC with only 8 frames, whereas this method processes video at 1 FPS, giving the two methods access to substantially different amounts of visual information. As Nar-KFC is one of the closest related methods, could the authors compare them using the same environment?

Q2: Query-guided memory writing can limit answers to subsequent questions.

The method mainly focus on a single active query, with a structural limitation for consecutive questions. By using the current query to select future keyframes, the method will drop visual memory that may be needed in subsequent queries. Switching queries or repeating retrieval cannot recover that evidence. This differs from those related works that compress memory independently of the query: even that they are evaluated on the same benchmarks, which test questions separately. In fact, continuous interaction is a basic setting in online video understanding, as studied in StreamBench, EgoServe, and early VideoLLM-online. Could the proposed method avoid this problem?

Q3: Does frame selection mainly benefit from qe rather than nov and gap?

In Table 7, qe alone achieves 63.3 overall, while adding both nov and gap only increases the score to 63.4. This suggests that the benefit mainly comes from qe. Can nov and gap scores actually preserve information needed for future, unknown questions? If not, why include them? Such strong dependence on the current query would also make the consecutive-question concern in Q2 more serious.

Q4: Why does performance decline across all FAR subtasks?

In Table 6, average FAR performance drops from 59.9 to 55.1, with declines on all three subtasks. This suggests that the memory mechanism may impair some streaming video understanding capabilities. The authors attribute this decline to the backbone receiving more complete video information. However, the backbone also directly accesses video in the other tasks. Why does FAR consistently deteriorate, particularly REC, which drops by 11.2 percentage points?

Q5: Can the system sustain polling once per second in practice?

The paper states that the VLM is polled once per second, whereas Table 4 reports a query TTFT of 5.76 seconds. Could this cause an accumulating latency between inference and the video stream? Could the actual delay from evidence arrival to answer delivery become several times larger than the reported TTFT?

References:

[1] Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders (Nar-KFC).

[2] Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (StreamChat / StreamBench). ICLR 2025.

[3] Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos (EgoServe). ECCV 2026.

[4] VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024.

9 comments
AuthorsOpenReview ↗

Response to Reviewer cChp (Part1)

Thank you for your constructive comments. Below are our point-by-point responses.


[W1.1 / Q1] Design differences from Nar-KFC and comparison under matched input limits

"W1: The advantages over existing methods are unclear. ... Therefore, it is difficult to determine whether the gains come from the proposed memory mechanism."

"Q1: Can the comparison with Nar-KFC use matched input budgets? ... could the authors compare them using the same environment?"

We thank the reviewer for suggesting this clarification and comparison. The different processing order reflects the information available in offline and streaming settings. Nar-KFC addresses offline long-video understanding, where the complete video and the query are available for keyframe selection. It can therefore first select query-relevant and diverse keyframes across the video, then use textual narratives to supplement the temporal content between them.

EviStream instead constructs memory as video frames arrive, before future frames or queries are available. ISM incrementally summarizes the observed content, and MEKS then uses the resulting semantic representation to guide keyframe retention through text-gap feedback and temporal novelty. Generating the summary first allows the system to assess which visual information is insufficiently represented in that summary and prioritize complementary frames.

Comparison with aligned input limits on offline datasets. The eight frames reported for Nar-KFC refer to its answer-time keyframes, rather than all frames considered during selection. Both methods use candidate frames sampled at 1 FPS in our comparison. Following the reviewer’s suggestion, we evaluate EviStream on Video-MME Long without subtitles, reducing its answer-time historical keyframe limit to eight and increasing its textual-entry limit from 30 to 210 to match Nar-KFC’s narrative count.

Table XI. Video-MME Long results with eight keyframes and up to 210 textual entries.

MethodVideo-MME Long (%) ↑
Nar-KFC52.0
EviStream (ours)56.4

EviStream outperforms Nar-KFC by 4.4 percentage points in this comparison, demonstrating competitive performance under the same frame and textual-entry limits. We will include these results and specify the backbone, inference settings, and input budgets for both methods in the revised manuscript.


[W1.2] Beyond visual redundancy reduction: comparison with TimeChat-Online and our core insight

"Moreover, methods such as TimeChat-Online compress similar visual content and retain differences. ... similar to the proposed “abstraction plus supplementation”, with the main difference being that this work summarizes shared content as text. "

We acknowledge that EviStream builds on visual redundancy reduction, as explored by TimeChat-Online and cited in Section 3.2. Visual-centric methods primarily reduce redundant visual tokens, while memory-centric methods such as StreamForest organize historical information to extend context coverage. EviStream integrates these directions through the coordinated construction of evolving semantic memory and complementary visual evidence.

Our key insight is that visual retention should depend on what the evolving semantic memory has already captured. ISM incrementally records new semantic information relative to the accumulated history, while MEKS uses text-gap feedback and temporal novelty to preserve visual details insufficiently represented in semantic summaries. This coordination enables semantic memory to guide visual evidence retention, maintaining complementary information across modalities. At query time, relevant semantic history and associated visual evidence are jointly retrieved to support subsequent reasoning.

AuthorsOpenReview ↗

Response to Reviewer cChp (Part2)

[W1.3] Comparisons with a matched backbone and frame rate

"More concerningly, the ... this work uses the stronger Qwen3.5-9B without comparing these methods under the comparable model... whether the gains come from the proposed memory mechanism."

We thank the reviewer for raising this concern. We chose Qwen3.5-9B to evaluate whether EviStream can improve an already strong backbone. However, we agree that comparisons across different backbones do not isolate the contribution of the memory framework.

We therefore add comparisons using the same Qwen3-VL-8B backbone and a fixed input frame rate of 1 FPS. Table 12 compares EviStream with the bare backbone, Response-G1, STRIDE, and HERMES under these settings.

Table XII. Comparisons with Qwen3-VL-8B.

MethodStreamingBench (%)OVO-Bench RT (%)OVO-Bench BT (%)OVO-Bench Avg. (%)
Qwen3-VL-8B56.467.245.756.5
+ Response-G177.573.652.162.9
+ STRIDE69.769.747.858.8
+ HERMES79.069.853.761.8
+ EviStream (ours)80.273.761.167.4
EviStream achieves the highest scores across the reported metrics under the matched backbone and frame rate. On StreamingBench, it improves over the bare backbone by 23.8 percentage points and exceeds the strongest alternative, HERMES, by 1.2 points. On the reported OVO-Bench average, it exceeds Response-G1 by 4.5 points, with a particularly pronounced advantage on BT.

These results show that EviStream’s gains are not solely attributable to the stronger Qwen3.5-9B backbone and support the effectiveness of the framework on Qwen3-VL-8B as well. We will incorporate these comparisons into the revised Tables 1 and 2 and document the evaluation settings.


[W2] Recovering visual evidence omitted from textual summaries

"W2: The retrieval logic at answer time is confusing ... it may remain inaccessible to the answering model."

We thank the reviewer for identifying this potential retrieval failure. Textual retrieval provides anchor windows, rather than restricting visual access to the directly retrieved entries. After locating relevant semantic memories and their source windows, EviStream applies Anchor Neighborhood Expansion (ANE) to include adjacent temporal windows and additionally retains the two most recent windows. The associated visual keyframes are recovered through shared window identifiers.

This allows a window to be accessed through the broader event context even when its summary omits a specific visual detail. Neighboring windows and recent observations provide additional routes to the relevant evidence. However, these mechanisms mitigate rather than eliminate the risk: if neither the relevant window nor a suitable neighboring anchor is retrieved, preserved visual evidence may remain inaccessible.

Our additional analysis in response to Q3 measures Recall@128 for visual evidence absent from semantic memory. Recall reaches 73.1% with NOV and GAP, indicating that the evaluated pipeline recovers a substantial proportion of the omitted evidence, while leaving a residual retrieval gap.

We will clarify the anchor expansion and recent-window inclusion mechanisms in the revised manuscript and explicitly discuss failures caused by relevant windows falling outside retrieval coverage.

AuthorsOpenReview ↗

Response to Reviewer cChp (Part3)

[Q2] Query-guided memory writing and subsequent questions?

"Q2: Query-guided memory writing can limit answers to subsequent questions ... Could the proposed method avoid this problem?"

We thank the reviewer for raising this concern. EviStream maintains a query-independent semantic history rather than constructing its entire memory around the active query. ISM continuously consolidates observed events independently of user queries. MEKS additionally preserves visual evidence using novelty and text-gap feedback before query arrival, with query relevance introduced as an additional selection signal once a query becomes available. Thus, later questions can draw on accumulated semantic history and retained visual evidence beyond the current query’s immediate needs. This mitigates query-specific information loss, although it cannot guarantee preservation of every visual detail needed by future questions.

Evaluation on consecutive, context-dependent questions. We additionally evaluate EviStream on StreamingBench’s Sequential Question Answering (SQA) task. SQA presents sequences of related questions over the same video, with later questions referring to entities or events introduced in earlier interactions. Answering these questions requires resolving cross-turn references and combining conversational context with the available visual evidence. Following the evaluation protocol, previous questions, answer options, and ground-truth answers are included as dialogue history.

Table XIII. Performance on StreamingBench SQA.

MethodSQA (%) ↑Gain (pp)
Qwen3.5-9B42.8—
STRIDE46.4+3.6
EviStream (w/o query evidence)50.4+7.6
EviStream58.8+16.0

Without query evidence, EviStream already exceeds the backbone by 7.6 percentage points, supporting the effectiveness of its query-independent memory configuration. Adding query evidence further improves accuracy by 8.4 points, reaching 58.8%. These results indicate that query-guided selection provides a net benefit under the evaluated sequential-question protocol, while query-independent memory remains useful across related questions.

We will include these results and clarify how query-independent memory and query-guided selection jointly support consecutive questions in the revised manuscript.

Reviewer_q3CROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents EviStream, a training-free memory framework for streaming video understanding. The main idea is to maintain two complementary forms of memory: a compact textual history of events and a sparse set of visual keyframes intended to preserve details lost during textual summarization. The method has three components: incremental semantic summarization over adaptive video windows, keyframe selection based on temporal novelty, query relevance, and disagreement with the text summary, and a continual polling mechanism that keeps updating and retrieving memory after a query arrives until the model decides it has enough evidence to answer.

Using a frozen Qwen3.5-9B backbone, EviStream improves StreamingBench real-time understanding from 78.3 to 84.1 and complete OVO-Bench performance from 58.6 to 63.4. The paper also reports reduced response-time GPU memory and latency. Ablations study window construction, post-query updates, keyframe selection signals, and retrieval budgets, although they do not fully isolate all components.

Critical strengths
  1. The semantic-to-visual feedback loop is a meaningful and plausibly novel idea. Rather than maintaining textual and visual memories independently, EviStream uses the generated semantic delta to help decide which raw frames may contain information missing from that abstraction. This is a well-motivated formulation of complementary memory and is more principled than retaining frames solely through recency or visual novelty. The shared window identifiers and provisional-to-formal memory updates also make the overall design internally coherent.

  2. The complete training-free system produces substantial same-backbone gains. With frozen Qwen3.5-9B, EviStream improves StreamingBench real-time understanding from 78.3 to 84.1 and complete OVO-Bench performance from 58.6 to 63.4, while retaining only a sparse set of historical frames. These controlled gains indicate that the memory wrapper provides real practical value beyond merely using a stronger backbone. The absence of task-specific training also makes the approach comparatively easy to adapt and deploy.

Critical weaknesses
  1. The central evidence-preservation mechanism is not convincingly validated. The paper’s main novel claim is that the text-gap score identifies visual information omitted by semantic memory. However, there is no direct evaluation of omitted-evidence recall, and the MEKS ablation compares selectors that retain very different numbers of frames. At approximately matched retention, query-only selection scores 63.3, while the full three-signal method scores 63.4. Thus, the experiments demonstrate that sparse frame selection can be useful, but not that the proposed text-gap mechanism preserves complementary evidence. This leaves the paper’s principal technical contribution largely unsubstantiated.

  2. The claimed scalability to real-time, unbounded streams is not demonstrated. Every textual delta and selected keyframe is retained, and each new summary conditions on the full accumulated textual history, so storage and update cost grow with stream duration. Moreover, the system claims one-second polling while reporting a 5.76-second time to first token, without explaining the required concurrency or total compute. The efficiency results measure only the final response call and exclude continual summarization, retrieval, storage, and repeated polling. These omissions undermine the paper’s central systems premise rather than representing minor implementation details.

Questions for authors
  1. What state is actually bounded over an unbounded stream? Please clarify how textual history, stored keyframes, cached embeddings, and summary-prompt length grow, and what happens when the backbone context limit is reached.
  2. How is one-second polling implemented given the reported 5.76-second TTFT? Please specify the concurrency schedule, GPUs required per active stream, achieved polling rate, and whether evaluation uses logical video time rather than wall-clock execution.
  3. Were the MEKS weights, thresholds, window bounds, and 128-frame budget selected using a held-out validation set or the reported test benchmarks?
  4. Can the authors provide a fixed-budget comparison in which query-only, novelty, text-gap, random, uniform, and full MEKS retain exactly the same number of frames? More importantly, does the text-gap score measurably retrieve frames containing facts absent from the generated summary?
  5. How does EviStream compare under the same Qwen3.5 backbone and resource budget with a recent-frames baseline, an uncoupled text-plus-keyframe memory, and the closest training-free alternatives such as SAVEMem or Visual Agentic Memory and similar baseline? These comparisons would materially affect whether I view the gains as evidence for EviStream’s specific design.
3 comments
AuthorsOpenReview ↗

Response to Reviewer q3CR (Part1)

We really appreciate your valuable comments and questions, which help us a lot to improve our work. We address your concerns as follows.


[W1 / Q4] Clarifying signal activation and the interpretation of the MEKS ablation

"The central evidence-preservation mechanism is not convincingly validated ... This leaves the paper’s principal technical contribution largely unsubstantiated."

" Can the authors provide a fixed-budget comparison in which query-only ... absent from the generated summary?"

We apologize for not clearly specifying when each MEKS signal is available. Across StreamingBench and OVO-Bench, query relevance can guide keyframe selection only after a query arrives in proactive-response tasks. Before query arrival, the system has no access to the future query, and historical keyframe selection relies on temporal novelty and text-gap feedback.

This distinction is important for interpreting Table 7. The aggregate comparison between 63.3 and 63.4 should not be taken as evidence that novelty and text-gap are unnecessary for query-free historical memory construction. We will clarify the signal-activation rules and explicitly describe how frames are retained when the enabled signals are unavailable.

Comparison under a fixed visual budget. To further evaluate the individual contributions of the selection signals, we conduct additional experiments on StreamingBench using Qwen3.5-9B with the same inference settings. We compare six strategies under a maximum budget of 128 historical keyframes per sample: Query Evidence, Novelty, Text-Gap, Random, Uniform, and Full MEKS. Random selection assigns pseudorandom scores to frames, while Uniform selection samples frames at evenly spaced intervals. Both strategies combine the selected frames with the same semantic memory for QA. To quantify visual evidence preservation, we additionally introduce Recall@128, computed by matching the timestamps of selected keyframes with query-relevant temporal evidence intervals. This metric measures the temporal coverage of relevant visual information that may be insufficiently represented in semantic summaries.

Table VIII. Comparison of keyframe selection strategies on StreamingBench.

ConfigurationSqeS^{qe}SnovS^{nov}SgapS^{gap}Recall@128 (%)RTVU (%)
Qwen3.5-9B––––78.3
Random––––79.2
Uniform––––81.0
EviStream (QE)✓✗✗0.080.4
EviStream (NOV)✗✓✗50.582.5
EviStream (GAP)✗✗✓69.883.6
EviStream (Full)✓✓✓73.184.1

The results demonstrate the effectiveness of Novelty and Text-Gap in preserving relevant visual information. Text-Gap achieves 69.8% Recall@128 and 83.6% RTVU accuracy, outperforming Novelty, Random, and Uniform selection. Full MEKS further achieves the highest Recall@128 of 73.1% and RTVU accuracy of 84.1%. These findings support the contribution of Text-Gap to preserving complementary visual evidence when future queries are unknown. Furthermore, Query Evidence is unavailable during historical memory construction in RTVU tasks, whereas its contribution becomes more pronounced in proactive response settings where the query is available before subsequent observations. We will include these comparisons and clarify the evidence recall protocol in the revised manuscript.


[W2.1 / Q1.1] Bounded processing budgets and the limits of long-term memory growth

"The claimed scalability to real-time, unbounded streams is not demonstrated. Every textual delta and selected keyframe is retained, and each new summary conditions on the full accumulated textual history, so storage and update cost grow with stream duration. "

"What state is actually bounded over an unbounded stream?"

We thank the reviewer for highlighting this distinction. EviStream bounds local visual processing and query-time historical frame input, but not total memory or update cost over unbounded streams.

ISM processes 20–30-frame windows with visual token pruning. At query time, semantic retrieval returns 30 initial candidates, and historical visual input is capped at 128 frames. These limits constrain local visual inputs, but neither stored history nor retrieval cost.

For long-term memory, ISM incrementally consolidates related event details and removes adjacent redundancy. This reduces repetition without limiting stored textual memory, keyframes, or cached embeddings. As history grows, summary prompts become longer, increasing update cost and potentially reaching the backbone’s context limit.

EviStream thus supports incremental processing of incoming video with limited local visual inputs and sparse access to historical evidence. It does not establish constant storage or update cost for indefinitely long streams. We will clarify this scope and discuss bounded long-term memory management as a limitation and future direction.

AuthorsOpenReview ↗

Response to Reviewer q3CR (Part2)

[W2.3 / Q1.2] Memory growth and the scope of efficiency measurements

"The efficiency results measure only the final response call and exclude continual summarization, retrieval, storage, and repeated polling. "

"Please clarify how textual history, stored keyframes, cached embeddings, and summary-prompt length grow, and what happens when the backbone context limit is reached."

We thank the reviewer for examining the systems implications of our method. We acknowledge that the costs of semantic summarization, memory storage, and related operations increase with video duration. To quantify long term resource growth, we report approximate averages aggregated across our completed experiments, as shown below.

Table IX. Memory growth across different video durations.

Video Duration (min)Textual History (k tokens)KeyframesEmbedding Cache (MB)Summary Prompt (k tokens)
1–103.7560.683.9
10–3013.04751.7513.3
30–6033.912404.5634.1

The results show that memory requirements increase with video duration while remaining manageable within the evaluated range, supporting the feasibility of our central systems premise.

Furthermore, the accumulated semantic history in our experiments remains within the context length limit of Qwen3.5-9B, so no history truncation is required. For longer video streams, we plan to impose a fixed token budget on ISM summarization prompts, prioritizing recent semantic increments and the current visual window while preserving older records in external memory for subsequent retrieval. We will discuss these limitations in the revised manuscript and identify the resource efficiency and stability of long term operation as directions for future work.


[W2.2 / Q2] Polling schedule and the distinction between video time and inference latency

"Moreover, the system claims one-second polling while reporting a 5.76-second time to first token, without explaining the required concurrency or total compute.

How is one-second polling implemented given the reported 5.76-second TTFT? Please specify the concurrency schedule, GPUs required per active stream, achieved polling rate, and whether evaluation uses logical video time rather than wall-clock execution.

We thank the reviewer for highlighting this distinction. EviStream employs asynchronous frame buffering and sequential inference, with the one-second polling interval defined as the response decision frequency in logical video time. When a query arrives, the system initiates the first polling decision based on the available evidence. Since the initial inference involves prefill of a long context, incoming frames are buffered in a queue during this process. The model subsequently processes queued frames in timestamp order, making a response or silence decision for each frame until sufficient evidence becomes available. Therefore, the 5.76-second TTFT reported in Table 4 measures the inference latency from query arrival to the first output token, which is distinct from the one-second polling interval in logical video time.

Our experiments use eight NVIDIA A800 GPUs with sample-level data parallelism, where each GPU worker independently processes one video stream. We will further clarify the scheduling mechanism, polling interval definition, and evaluation protocol in the revised manuscript.


[Q3] Hyperparameter selection protocol

Were the MEKS weights, thresholds, window bounds, and 128-frame budget selected using a held-out validation set or the reported test benchmarks?

We thank the reviewer for their valuable feedback regarding our hyperparameters. StreamingBench and OVO-Bench do not provide dedicated validation splits under our evaluation protocols. Following the empirical hyperparameter exploration practices of prior training-free methods such as HERMES, we conducted multiple comparative experiments on the publicly available benchmarks and adopted a fixed set of hyperparameters across all reported evaluations.

Specifically, the MEKS weights and selection threshold were determined through repeated comparisons of accuracy and frame retention. The adaptive window bounds [20, 30] were selected based on the sensitivity analysis in Figure 3, while the maximum budget of 128 retrieved keyframes was determined through the accuracy and truncation ratio analysis in the appendix. We will further clarify the hyperparameter selection procedure and provide additional details on the evaluated configurations and parameter sensitivity in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer q3CR (Part3)

[Q5] Comparisons with recent-frame, uncoupled-memory, and training-free baselines

How does EviStream compare under the same Qwen3.5 backbone and resource budget ... as evidence for EviStream’s specific design.

We thank the reviewer for suggesting these comparisons. We evaluate three alternatives using the same frozen Qwen3.5-9B backbone and a maximum visual input budget of 128 frames: (A) SimpleStream as a recent-frames baseline; (B) an uncoupled text-plus-keyframe variant of EviStream that retains semantic memory but selects keyframes using novelty alone; and (C) SAVEMem as a related training-free memory method. The bare backbone uses uniformly sampled frames.

For these experiments, we set the frame budgets of SimpleStream and SAVEMem to 128, matching EviStream’s maximum visual input budget. This controls the number of input frames, but does not imply identical total computation or memory overhead across methods.

Table X. OVO-Bench results with Qwen3.5-9B and a matched maximum visual input budget.

MethodRT (%) ↑BT (%) ↑FAR (%) ↑Overall (%) ↑
Qwen3.5-9B69.846.259.958.6
(A) SimpleStream69.252.454.958.8
(B) EviStream (w/o Align)76.157.246.960.1
(C) SAVEMem79.845.144.056.3
EviStream (ours)76.858.455.163.4

EviStream achieves the highest overall score, exceeding SimpleStream, the uncoupled-memory variant, and SAVEMem by 4.6, 3.3, and 7.1 percentage points, respectively. Compared with the uncoupled variant, full EviStream improves RT by 0.7 points, BT by 1.2 points, and FAR by 8.2 points. This comparison supports the contribution of the coupled selection design beyond simply combining textual memory with novelty-selected keyframes.

The results also reveal task-specific trade-offs. SAVEMem achieves the highest RT score, exceeding EviStream by 3.0 points, while the bare backbone remains strongest on FAR. EviStream’s advantage is therefore in overall performance under the evaluated visual input budget, rather than uniform superiority across all task groups.

We will include these comparisons and the baseline adaptations in the revised manuscript, and explicitly distinguish the matched visual input budget from end-to-end resource cost.

Reviewer_Tc4hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EviStream is a streaming video framework that requires no training and runs on a frozen Qwen3.5-9B. Before a query arrives, Incremental Semantic Memory segments the 1 fps stream into adaptive windows of 20 to 30 frames, prunes temporally redundant tokens following TimeChat-Online, and has the model write a textual delta for each window conditioned on the accumulated history. Memory-Aligned Evidence Keyframe Selection keeps a sparse set of keyframes per window using query relevance, patch retention novelty, and a text gap score that measures the distance between a frame’s mean token embedding and the embedding of the delta text. After the query, Continual Evidence-Polling Reasoning retrieves memory lines by hybrid reciprocal rank fusion with neighbourhood expansion, recovers the linked keyframes, polls the model once per second for a silence or respond decision, and keeps consolidating new and provisional windows while waiting. Relative to the backbone, StreamingBench real time accuracy rises from 78.3 to 84.1 and PO from 21.6 to 47.6. On OVO-Bench, RTVP rises from 69.8 to 76.8, BT from 46.2 to 58.4 and overall from 58.6 to 63.4, while FAR falls from 59.9 to 55.1. LongVideoBench is 63.6 and Video-MME 66.4, with lower peak memory and TTFT than the backbone. Ablations cover window strategy and bounds, the three selection signals, memory evolution after the query, and the keyframe budget.

Critical strengths
  1. The system works and the comparison with its own backbone is clean. It gains 5.8 points on StreamingBench real time understanding, 26 points on PO and 4.8 points on OVO overall, while peak memory falls from 21.73 to 19.24 GB and TTFT from 6.31 to 5.76 s. Keyframe selection as a whole is useful: dropping 83.5 percent of historical frames keeps RTVP and BT unchanged and raises FAR by 8.7 points over no selection. The appendix gives the full algorithm, the retrieval formulation, the provisional update rules and all hyperparameters, which is enough to reimplement the system.
  2. The framing is sound. Which visual evidence should survive textual abstraction, and how memory should keep evolving after a query, are real gaps in streaming systems.
Critical weaknesses

1.The two mechanisms presented as contributions are not validated on their own. In the keyframe selection ablation, query relevance alone reaches 63.3 overall and 58.4 on BT. Adding novelty and the text gap score gives 63.4 and 58.4 at nearly the same retention, so the new signals add at most 0.1 point. For memory evolution after the query, the ablation gives 45.9 PO with memory frozen at the query and 47.6 with full evolution. This 1.7 point difference is small against the 26 point total gain over the backbone, and the paper provides no analysis stratified by the interval between query and evidence to support its introduction’s claim that the benefit grows with that interval. 2.Attributions and claim scope go beyond the evidence. The BT gain of 12.2 points comes entirely from HLD, which rises from 8.6 to 57.5, while EPM falls from 59.6 to 54.9 and ASI from 70.3 to 62.8. HLD rewards declining to answer, and the OVO-Bench paper reports a text only GPT-4-turbo with no video input at 70.05 on it, so the jump may reflect a shift toward abstention rather than better recall. The paper does not test this, and its attribution of the BT gain to keyframe selection is unsupported. Counting regresses on both benchmarks, CT from 55.9 to 52.1 and REC from 50.4 to 39.2, although counts are the detailed content that keyframe selection is meant to preserve. These regressions are not discussed, and the appendix attributes the FAR drop to offline simulation without a test. Comparisons with prior methods use different backbones and sampling rates. Response-G1 runs on Qwen3-VL-8B with sampling between 0.2 and 1 fps, while EviStream runs on Qwen3.5-9B at a fixed 1 fps, and the bare Qwen3.5-9B already scores 78.3 against 77.5 for Response-G1, so the 6.6 point margin mixes backbone, sampling and method. The offline results omit the bare backbone. The official Qwen3.5-9B model card reports 78.4 on Video-MME without subtitles, and this paper reports 66.4 under its own settings, so the offline claim cannot be interpreted without a baseline under identical settings.

Questions for authors

1.Selection Signals and Memory Evolution. Please separately evaluate evidence preservation before the query, where only novelty and text gap are active, and query aware selection afterward, reporting their frame retention rates and their contributions to downstream QA accuracy. For the memory evolution ablation, please specify the number of PO examples and the annotation version, report paired uncertainty estimates, and analyze whether the benefit increases with the interval between query arrival and answerable evidence. 2.HLD and counting. Report how often the backbone and EviStream decline to answer on answerable BT and RTVP questions, and add a control that uses textual memory with no keyframes on HLD. Explain why CT and REC regress. 3.Backbones and offline baselines. Run Response-G1 or HERMES on Qwen3.5-9B, or EviStream on Qwen3-VL-8B, under the same frame rate. Add the bare Qwen3.5-9B to the offline results with the same frame budget and inference mode as EviStream.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tc4h (Part1)

Thank you for the detailed and constructive feedback. We address your concerns below.


[W1 / Q1] Selection Signals and Memory Evolution

"1.The two mechanisms presented as contributions are not validated on their own."

"1.Selection Signals and Memory Evolution."

[W1.1 / Q1.1] Selection Signals: Interpreting Table 7 and Additional Pre-/Post-Query Experiments

"In the keyframe selection ablation, query relevance alone reaches 63.3 ... new signals add at most 0.1 point."

"Please separately evaluate evidence preservation ... to downstream QA accuracy. "

We thank the reviewer for requesting this separation, a concern also raised by Reviewers q3CR and cChp. We agree that the aggregate comparison in Table 7 does not sufficiently isolate the contributions of novelty and text-gap. We clarify the signal-activation rules and provide additional stage-specific experiments below.

Roles and availability of the three selection signals. Temporal novelty prioritizes visual changes relative to preceding observations, while text-gap prioritizes visual content that may be insufficiently represented in semantic memory. Together, they support evidence preservation for unknown future questions. Query relevance instead helps select evidence for an already available question.

Query relevance is not available in every memory-construction setting. In non-proactive tasks, historical memory is constructed before the question is revealed, so keyframe selection relies only on novelty and text-gap, as query relevance is unavailable. The same restriction applies before query arrival in proactive-response tasks. In proactive setting, only after a query arrives, while the system continues observing the stream and waiting for sufficient evidence, can query relevance additionally guide subsequent selection. Accordingly, S_qe is set to zero before the query timestamp and is activated only in FAR in our OVO-Bench evaluation.

The 63.3-versus-63.4 comparison therefore does not separately establish the effectiveness of query-independent historical selection and query-guided post-query selection. Following the reviewer’s suggestion, we examine these configurations separately.

Pre-query versus post-query evidence on FAR. Table I reports frame retention and downstream FAR accuracy. The post-query variant begins retaining visual frames only after query arrival.

Table I : Stage-specific visual evidence configurations on FAR.

ConfigurationRetention (%)FAR accuracy (%)
w/o Vision046.4
Pre-query: Novelty + Text-Gap6.147.2
Post-query: Query Evidence only17.055.0

Relative to the configuration without visual evidence, pre-query NOV/GAP selection improves FAR accuracy by 0.8 percentage points, while query-guided post-query selection improves it by 8.6 points. The larger post-query gain is consistent with FAR requiring newly arriving evidence for an active question. These configurations differ in temporal coverage and retention rate, so the comparison characterizes their roles rather than ranking the scoring signals under identical inputs.

NOV and GAP remain essential for preserving historical evidence before future queries become available: QE cannot replace them when future questions are unknown. We therefore additionally evaluate their contribution in query-independent memory construction.

Evidence preservation before unknown queries. Table II reports results on StreamingBench’s contextual understanding tasks, ACU and MCU, and its event understanding and clip summarization tasks, EU and CS. Historical keyframes are selected without access to the future query.

Table II. Query-independent evidence selection on StreamingBench.

MethodSqeS^{qe}SnovS^{nov}SgapS^{gap}Recall@128 (%)ACU (%)MCU (%)EU (%)CS (%)
STRIDE————32.038.476.477.3
Qwen3.5-9B————38.832.081.785.8
EviStream✗✗✗053.642.881.489.6
EviStream✗✓✓73.160.843.682.491.8

Enabling NOV and GAP without QE improves all four task scores, by 7.2 points on ACU, 0.8 on MCU, 1.0 on EU, and 2.2 on CS, while achieving 73.1% Recall@128. These results support their joint contribution to retaining useful visual evidence before the query is known.

Together, the analyses support complementary roles across the streaming timeline: NOV and GAP support query-independent historical memory, while QE provides additional guidance for observations arriving after an active query. We will include these experiments and explicitly document the activation rules, ablation configurations, and retention and recall definitions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Tc4h (Part2)

[W1.2 / Q1.2] Post-query memory evolution: overall PO gain and stratified analysis by query-to-evidence interval

"For memory evolution after the query, the ablation gives 45.9 PO ... claim that the benefit grows with that interval. "

"For memory evolution after the query, the ablation gives 45.9 PO ... that the benefit grows with that interval. "

Overall PO gain. We agree that the 1.7-point gain from post-query memory evolution is modest. Freezing memory at query arrival yields a PO score of 45.9, and continuing to update memory raises it to 47.6. This ablation measures the incremental benefit of continued updates within EviStream. The 26.0-point improvement over the backbone reflects the performance of the complete framework, which combines memory construction, evidence selection and retrieval, and response decisions.

Stratified analysis by query-to-evidence interval. We thank the reviewer for suggesting this analysis. Following the reviewer’s suggestion, we group PO examples by Δt=te−tq\Delta t = t_{e} - t_q, where tqt_q is the query arrival time and tet_{e} is the earliest time at which sufficient evidence is available to answer. Within each group, we compare the PO dataset scores between the Full Memory Evolution and Frozen Memory variants under the same settings.

Table III. PO scores by query-to-evidence interval.

Query-to-evidence intervalFrozen Memory (%)Full Memory Evolution (%)Gain (pp)
Short (0–5 s)54.956.0+1.1
Medium (5–20 s)42.043.6+1.6
Long (>20 s)49.754.9+5.2
Overall45.947.6+1.7

The PO gain from memory co-evolution increases with the query-to-evidence interval, from +1.1 pp for short intervals to +1.6 pp for medium intervals and +5.2 pp for long intervals. This pattern supports our claim that continued memory updates are especially helpful when the evidence needed to answer takes longer to arrive. In real-world streaming video understanding, the arrival time of sufficient evidence is unknown when a query is posed. EviStream addresses this uncertainty through memory co-evolution: it continually consolidates new observations and refreshes retrieval while the query remains pending, allowing accumulated evidence to inform subsequent response decisions.


[W2 / Q2 / Q3] Attributions and claim scope

"Attributions and claim scope go beyond the evidence."

[W2.1 / Q2.1] HLD-driven BT gains: abstention behavior and the contribution of keyframes

"The BT gain of 12.2 points comes entirely from HLD, which rises from 8.6 to 57.5, ... is unsupported. "

"HLD and counting. Report how often the backbone ... no keyframes on HLD. "

We thank the reviewer for distinguishing appropriate abstention from visual recall. The aggregate BT improvement is driven by HLD, while EPM and ASI decline. HLD evaluates whether the model selects the abstention option for questions unsupported by the video. Its accuracy therefore reflects correct abstention on these examples, rather than directly establishing improved visual evidence recall.

GPT-4-turbo is a closed-source model whose API is no longer available. The original text-only evaluation details are also insufficient for reproduction, and recent studies, including ours, have been unable to reproduce the reported HLD score. While this limits direct verification, the result remains relevant to the possibility of language-only cues. To assess the contribution of visual memory within EviStream, we therefore provide a controlled comparison using the same backbone with and without keyframes.

Contribution of keyframes. We add a control retaining textual memory while removing keyframes.

Table IV. HLD accuracy with and without keyframes.

ConfigurationHLD accuracy (%)
EviStream without keyframes56.1
EviStream (Full)57.5

Adding keyframes yields an observed improvement of 1.4 percentage points. However, the variant without keyframes already achieves 56.1%, compared with the backbone’s 8.6%. Most of the HLD improvement therefore persists without keyframes, and we will remove the attribution of the aggregate BT gain to keyframe selection.

EPM and ASI regressions. Regressions on EPM and ASI have also been reported for methods that select or compress historical visual information, including HERMES and ShallowStream. This suggests that preserving fine-grained historical evidence under compression remains challenging. The backbone directly processes sampled video frames, whereas EviStream reduces memory overhead through visual redundancy pruning and semantic consolidation. This process may omit details needed for EPM and ASI, providing a plausible explanation for their lower scores. We will discuss this trade-off and directions for improving historical evidence preservation in the revised Limitations and Future Work section.

AuthorsOpenReview ↗

Response to Reviewer Tc4h (Part3)

[W2.2 / Q2.2] Counting regressions on CT and REC

"Counting regresses on both benchmarks, CT from 55.9 to 52.1 ... These regressions are not discussed, "

"Explain why CT and REC regress. "

We thank the reviewer for raising this issue. To investigate why EviStream underperforms the backbone on CT and REC, we ablate visual redundancy pruning and MEKS separately while holding the input frame rate, backbone, and inference settings fixed.

Table V. Effects of visual pruning and MEKS on counting and efficiency.

ConfigurationAGM (GB) ↓TTFT (s) ↓CT (%) ↑REC (%) ↑
w/o MEKS19.23.8951.137.4
w/o Visual Pruning21.06.2153.644.9
EviStream (Full)19.25.7652.139.2

Disabling visual pruning recovers 1.5 points on CT and 5.7 points on REC, while increasing Average GPU Memory from 19.2 to 21.0 GB and TTFT from 5.76 to 6.21 s. This ablation identifies visual pruning as one contributor to the counting regressions. A plausible explanation is that frames or tokens that appear redundant can still contain evidence needed to distinguish individual objects or separate occurrences of an action. The remaining gap to the backbone on both tasks suggests that disabling pruning alone is insufficient to recover the full counting performance.

MEKS helps preserve counting evidence within the compressed system. Removing MEKS lowers CT by 1.0 point and REC by 1.8 points. Together, the two ablations show that visual pruning trades away some counting accuracy for lower memory and latency, and that complementary keyframes improve counting relative to the variant without MEKS.

Similar counting regressions have been observed with other visual compression methods. Table 2 of Accelerating Streaming Video Large Language Models via Hierarchical Token Compression reports that adding other classical visual compression methods such as ToMe or VisionZip to ReKV reduces CT accuracy from 43.6% to 19.7% or 33.3%, respectively. We will include our ablations in the revised manuscript and discuss the challenge of preserving exact counts under a fixed compute budget in the limitations section.


[W2.3] Effect of evaluation settings on FAR performance

"and the appendix attributes the FAR drop to offline simulation without a test. "

We thank the reviewer for highlighting the need to test this attribution, a concern also raised by Reviewer cChp. The backbone’s offline simulation allows it to access the preceding video at each predefined timestamp, whereas EviStream relies on historical memory maintained as the stream progresses.

We therefore additionally evaluate EviStream under the backbone’s offline protocol. At each predefined timestamp, we construct the inference context from the video history available up to that point and directly perform response inference, without continuous memory evolution. We retain the same memory budget as in EviStream’s streaming setting.

Table VI. FAR performance under offline simulation and online streaming.

SettingREC (%) ↑SSR (%) ↑CRR (%) ↑Overall (%) ↑
Qwen3.5-9B (Offline)50.464.465.059.9
EviStream (Offline Sim.)44.667.768.160.1
EviStream (Streaming)39.262.763.355.1

Offline simulation raises EviStream’s overall FAR score from 55.1% to 60.1%, improving all three subtasks: +5.4 percentage points on REC, +5.0 on SSR, and +4.8 on CRR. Under the offline protocol, EviStream achieves a comparable overall score to the backbone (60.1% vs. 59.9%), exceeding it by 3.3 points on SSR and 3.1 points on CRR. REC remains 5.8 points lower, consistent with the counting limitations discussed in our response to W2.2 / Q2.2.

These results support evaluation setting as a contributing factor to the FAR gap, while the remaining REC deficit shows that it does not explain every subtask difference. The comparison also highlights the challenge of preserving event information for proactive responses through continuously maintained memory, rather than reconstructing context from the available video history at each decision point. We will add this experiment and its protocol to the revised appendix and revise the original attribution to reflect the scope of this evidence.

AuthorsOpenReview ↗

Response to Reviewer Tc4h (Part4)

[W2.4 / Q3.1] Comparison under a matched backbone and frame rate

"Comparisons with prior methods use different backbones and sampling rates ... so the 6.6 point margin mixes backbone, sampling and method. "

"Backbones and offline baselines. Run Response-G1 or HERMES on Qwen3.5-9B, or EviStream on Qwen3-VL-8B, under the same frame rate. "

We agree that the original 6.6-point margin over Response-G1 combines differences in backbone, sampling strategy, and method, and therefore cannot be attributed solely to EviStream.

Sampling protocol. EviStream processes incoming frames sequentially in windows and maintains a bounded historical memory, allowing it to process long streams at a fixed input rate of 1 fps. Methods that construct a video context at each query timestamp may instead adjust sampling according to video length and context constraints. These protocol differences motivate a comparison with a matched backbone and input frame rate.

Matched comparison. We evaluate EviStream with Qwen3-VL-8B and compare it with the bare backbone and Response-G1 at a fixed input rate of 1 fps.

Table VII. StreamingBench results with Qwen3-VL-8B.

MethodRTVU (%) ↑
Qwen3-VL-8B56.4
+ Response-G177.5
+ EviStream (ours)80.2

Under this matched setting, EviStream outperforms Response-G1 by 2.7 percentage points and improves over the bare backbone by 23.8 points. These results show that EviStream’s advantage persists with the same backbone and input frame rate, providing evidence for its contribution beyond the stronger Qwen3.5-9B backbone used in our main experiments.

We will add this comparison and its evaluation settings to the revised manuscript and distinguish the original cross-configuration comparison from the matched comparison reported here.


[W2.5 / Q3.2] Clarifying the matched offline baseline and the gap to the model-card score

"The offline results omit the bare backbone. The official Qwen3.5-9B ... without a baseline under identical settings. "

"Add the bare Qwen3.5-9B to the offline results with the same frame budget and inference mode as EviStream. "

We apologize for the unclear presentation. Our reproduced bare Qwen3.5-9B achieves 65.8% on Video-MME under the same frame budget and inference mode as EviStream (66.4%). The requested matched backbone baseline is therefore provided, and our comparison controls for these settings. We will clarify the baseline’s identity and evaluation configuration in the revised manuscript.

The gap between our reproduced backbone score (65.8%) and the model-card score (78.4%) likely reflects differences in inference configurations. A similar discrepancy was reported in official QwenLM Issue #86, “Weird evaluation results on Video-MME for Qwen 3.5-9B with Huggingface transformers,” where the issue reporter obtained 69.81%, substantially below the official score. Although this report does not establish the cause of our gap, it reinforces the importance of comparing methods under matched settings, as done in our experiments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the Area Chair and all reviewers for their time and constructive feedback. We apologize for the delayed response, as we needed additional time to complete and verify the controlled experiments addressing the reviewers’ concerns. We appreciate the reviewers’ recognition of EviStream’s motivation, memory design, and practical value:

  • An important and well-motivated problem (Tc4h, cChp). Reviewer Tc4h identified evidence preservation through textual abstraction and post-query memory evolution as “real gaps in streaming systems.” Reviewer cChp emphasized the practical importance of reducing visual computation while preserving evidence for future questions.
  • A promising and coherent memory design (q3CR). Reviewer q3CR described the semantic-to-visual feedback loop as “a meaningful and plausibly novel idea,” recognizing the rationale for using semantic summaries to guide complementary visual retention and the coherence of the memory-update design.
  • Meaningful same-backbone gains (Tc4h, q3CR). Reviewer Tc4h noted that “the system works and the comparison with its own backbone is clean.” Reviewer q3CR recognized the gains with sparse historical frames as providing “real practical value beyond merely using a stronger backbone.”
  • A training-free framework with clear presentation (all reviewers). All three reviewers rated the presentation as clear and easy to follow. The reviews also recognized the absence of task-specific training and the implementation details provided in the appendix.

The reviewers’ shared concerns primarily focus on empirical validation of the proposed mechanisms, controlled experimental comparisons, and clarification of implementation and evaluation details. We address these concerns through additional experiments, targeted analyses, and clearer explanations of the design and its relationship to prior work.

  1. Contributions of the selection signals and post-query memory evolution. We clarify that query relevance is unavailable during historical memory construction before a query is revealed. In our OVO-Bench evaluation, it is activated only for post-query observations in FAR. We therefore supplement the aggregate ablation with separate pre-/post-query analyses and query-independent selection experiments. For memory evolution, we separate its incremental contribution from the full-system gain and report results stratified by the query-to-evidence interval.

  2. Fair comparisons and interpretation of task-level results. We add comparisons using the same backbone and frame rate, together with input-budget controls. We distinguish HLD-driven BT gains from visual recall, add a keyframe control for HLD, and investigate counting regressions and the FAR gap through pruning ablations and offline-versus-streaming comparisons.

  3. Retrieval, continuous interaction, and operating limits. We clarify how retrieved anchors, neighboring windows, and recent observations provide access to visual evidence, and add sequential-question evaluation. We also report memory-growth statistics and distinguish logical video-time polling from wall-clock inference latency, clarifying which processing budgets are bounded and which memory states and costs grow with stream duration.

We acknowledge that the original submission did not make several experimental settings and implementation choices sufficiently explicit. The responses below provide additional evidence, clarify these details, and specify where the claims will be refined.

Response organization. We use W for comments under Critical Weaknesses and Q for Questions For Authors. Decimal numbering separates distinct points within a comment. Closely related weaknesses and questions are answered together, as in [W1.1 / Q1.1]. Rebuttal tables use Roman numerals to distinguish them from tables in the original submission.

We are carefully addressing all feedback and are confident that these clarifications and new results have addressed the reviewers' primary concerns. We will incorporate these updates into the revised version.

AuthorsOpenReview ↗

Apologies for the Delay: A Gentle Follow-up on Our Rebuttal Responses

Dear Reviewers and Area Chair,

Thank you for your time and consideration. We sincerely apologize for the delay in providing our responses, as we conducted extensive additional experiments to carefully address your comments and verify the results. We would greatly appreciate it if you could review our responses when your schedule permits. We sincerely look forward to your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.