acceptodds
Under review as a conference paper at ICLR 2027

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

Abstract

Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to Streaming video understanding requires multimodal large language models to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-andinternalize. Specifically, LatentStream comprises three coordinated components. First, Query-Agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into compact latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of the-art results on existing online and offline video benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Multimodal Large Language Models, Latent Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J19jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LatentStream, a training-free framework for streaming video understanding. It maintains a hierarchical visual memory and uses a small set of latent memory tokens to iteratively retrieve and internalize query-relevant historical evidence. A confidence-based objective optimizes these tokens at test time while keeping the backbone frozen. Experiments on online and offline video benchmarks show improvements over the base models.

Critical strengths
  1. The paper addresses a practical problem: improving streaming video question answering under a fixed memory budget without updating the backbone parameters.
  2. Combining historical evidence retrieval with test-time optimization of latent tokens is an interesting idea.
  3. The experiments cover both online and offline tasks and show performance gains across these settings.
Critical weaknesses
  1. Missing key baselines. The paper does not compare against SimpleStream [1] or SAVEMem [2], the latter already cited in the paper. A comparison with a simple recent-frame window under matched settings would help establish whether the additional retrieval and optimization mechanisms are necessary.
  2. Limited evidence of broader memory improvements. On Qwen2.5-VL-7B, the gain in Backward Tracing comes almost entirely from hallucination detection, with little improvement in episodic memory or action sequence identification (Table 1). The higher average score therefore does not establish a broad improvement in long-term memory.
  3. Insufficient validation of the “internalization” mechanism. The model still accesses the original visual memory bank during final answer generation (Sec. 3.4). The ablations in Table 4 show that optimizing latent tokens helps, but do not distinguish whether these tokens store historical information or simply help the model read the existing memory more effectively.

[1] Yujiao Shen, Shulin Tian, Jingkang Yang, and Ziwei Liu. A Simple Baseline for Streaming Video Understanding. arXiv:2604.02317, 2026.

[2] Hang Wu, Sherin Mary Mathews, Yujun Cai, Ming-Hsuan Yang, and Yiwei Wang. Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding. arXiv:2605.07897, 2026.

Questions for authors

Would the method still provide substantial gains with a stronger backbone, such as Qwen3.5 or a larger model? It would be useful to understand whether the benefits persist as the backbone’s capabilities improve.

6 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer J19j [Part 1]

We sincerely thank Reviewer J19j for the insightful comments. We address each point in detail below, and all corresponding revisions have been incorporated into the updated manuscript and highlighted in red. All references (e.g., Tables, Sections) refer to the revised version.


W1: Comparison with Key Baselines.

A: Thank you for pointing out these two relevant works. To solidly address your concern, we add comparisons with SimpleStream [ref1] (using the most recent eight frames) and SAVEMem [ref2] and conduct an additional experiment to examine whether our latent memory evolution strategy improves a simple recent-frame baseline.

(1) Comparison with SimpleStream and SAVEMem. The table below summarizes the published results of these baselines alongside the results from our original submission. For brevity, we report category-level and overall accuracy here; complete task-wise results are included in the revised manuscript.

MethodOVO RTOVO BWOVO OverallStreamingBench
Qwen2.5-VL-7B63.344.754.073.9
+ SimpleStream76.650.863.779.1
+ SAVEMem74.950.462.776.0
+ LatentStream (Ours)68.560.064.276.9
Qwen3-VL-8B70.553.161.873.2
+ SimpleStream79.954.967.478.8
+ LatentStream (Ours)80.467.373.882.0

As seen, with Qwen2.5-VL-7B, our LatentStream outperforms SimpleStream and SAVEMem by 0.5 and 1.5 points on OVO-Bench overall accuracy, respectively, with larger gains of 9.2 and 9.6 points on Backward Tracing (BW). With Qwen3-VL-8B, our LatentStream improves upon SimpleStream by 6.4 points on OVO-Bench overall, 12.4 points on OVO-Bench BW, and 3.2 points on StreamingBench. These results support the benefit of our approach, particularly for questions requiring historical information.

We also acknowledge that SimpleStream achieves better real-time perception (RT) and StreamingBench performance with Qwen2.5-VL-7B. We hypothesize that this difference may reflect how the two methods organize visual context. SimpleStream uses only the most recent eight frames, concentrating on recent visual details. In contrast, our LatentStream additionally incorporates historical memory and optimized latent memory to support reasoning. For questions primarily concerning the current scene, this additional context may divert attention from critical visual cues.

(2) Applying our latent memory evolution strategy to SimpleStream. To directly examine its contribution under the same recent-frame input, we incorporate our strategy into SimpleStream with Qwen3-VL-8B. Specifically, we retain the same eight-frame window and use its visual tokens as a fixed evidence pool. At query time, we iteratively retrieve relevant evidence from this pool to optimize a small set of Latent Memory Tokens (LMTs) through confidence-guided test-time optimization. Both variants access exactly the same eight frames, with no access to observations outside this window. The results under the same evaluation settings are summarized below.

MethodOVO RTOVO BWOVO OverallStreamingBench
SimpleStream79.954.967.478.8
+ Our Latent Memory Strategy (Ours)82.158.770.480.9

As seen, incorporating our latent memory evolution strategy improves SimpleStream by 3.0 and 2.1 points on OVO-Bench and StreamingBench, respectively. These results support the added value of our retrieval and latent memory optimization mechanisms beyond a simple recent-frame window and demonstrate that our strategy can also be extended to recent-frame window methods.

We have incorporated the baseline comparisons, the experiment applying our strategy to SimpleStream, and the corresponding discussions into Appendix Table 15, Table 11 and Sec. D.7 of the revised manuscript.

[ref1] A simple baseline for streaming video understanding. Arxiv 2026.

[ref2] Semantic-aware adaptive visual memory for streaming video understanding. Arxiv 2026.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer J19j [Part 2]

W2: Evidence of broader memory improvements.

A: Thank you for your thoughtful comment. We would like to clarify that, alongside the substantial HLD improvement (23.7→68.3), our LatentStream also improves ASI from 58.8 to 60.8 (+2.0 points) on Qwen2.5-VL-7B, indicating better performance in identifying historical action sequences. EPM shows a small decrease from 51.5 to 50.8. Thus, while HLD contributes the largest gain on this backbone, the benefits also extend to action sequence identification, with the magnitude of improvement varying across memory subtasks.

Our original submission further reports improvements in both EPM and ASI on Qwen3-VL-8B in Appendix Table 15. For clarity, we reproduce the relevant results below.

MethodEPMASIHLDBW Avg.
Qwen3-VL-8B52.560.146.853.1
+ OASIS62.060.147.357.2
+ LatentStream (Ours)64.364.972.667.3

As seen, LatentStream improves EPM and ASI over the backbone by 11.8 and 4.8 points, respectively, and outperforms OASIS by 2.3 and 4.8 points. Even when HLD is excluded, the mean accuracy across EPM and ASI increases from 56.3 to 64.6 (+8.3 points). These task-specific results provide direct evidence of a broad improvement in long-term memory on Qwen3-VL-8B.

The offline evaluations provide complementary evidence of improved long-video understanding. As reported in Table 2, LatentStream improves Qwen2.5-VL-7B by 3.3, 6.1, and 1.4 points on VideoMME, MLVU, and LongVideoBench, respectively.

We revise the OVO-Bench discussion in Section 4.2 to explicitly highlight the EPM and ASI improvements on Qwen3-VL-8B and their support for broader memory benefits.

W3: Validation of the internalization mechanism.

A: Thank you for pointing out this important distinction. Table 4 demonstrates the benefit of optimizing Latent Memory Tokens (LMTs). To directly examine whether the optimized Latent Memory Tokens (LMTs) store historical information or primarily help the model use the existing visual memory, we conduct an additional ablation below by removing the entire Hierarchical Streaming Memory (HSM) after LMT optimization. Specifically, we keep the optimization procedure unchanged, while generating the final answer using only the query and evolved LMTs. This setting tests whether information acquired during optimization remains usable through the LMTs after the original visual memory is removed.

MethodOVO-BenchVideoMME
HSM Only58.165.1
HSM + Retrieved Visual Evidence59.765.6
HSM + Initial Latent Memory Tokens56.565.2
Evolved Latent Memory Tokens Only (w/o HSM)63.666.2
HSM + Evolved Latent Memory Tokens (Ours)64.266.6

As seen, the evolved-LMT-only variant achieves 63.6% and 66.2% on OVO-Bench and VideoMME, respectively, outperforming HSM + Retrieved Visual Evidence by 3.9 and 0.6 points. Since the final decoder has no access to HSM in this setting, these results support the conclusion that the evolved LMTs store historical information relevant to the current query, rather than merely facilitating access to the existing memory. Furthermore, the full method improves over the LMT-only variant by 0.6 and 0.4 points, respectively, supporting the complementary contributions of evolved LMTs and HSM. These results suggest that LMTs serve as compact, query-conditioned working memory that retains historical information, while access to HSM provides additional visual context for answering.

We have incorporated this additional ablation into Table 4 and clarified the role of LMTs and their relationship with HSM in Sec. 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer J19j [Part 3]

Q1: Consistent gains with stronger backbones.

A: Good comment! Following your suggestion, we conduct additional experiments on OVO-Bench with Qwen3.5-9B to examine whether our LatentStream continues to provide substantial gains as backbone capabilities improve. Both variants use the same frame sampling and visual input budget.The task-wise results are as follows.

MethodOCRACRATRSTUFPDOJRRT Avg.EPMASIHLDBW Avg.Overall
Qwen3.5-VL-9B83.962.470.746.176.270.668.361.662.859.161.264.8
+ LatentStream (Ours)95.383.584.573.077.283.282.865.766.274.268.775.7

As shown above, our LatentStream improves overall accuracy from 64.8% to 75.7% (+10.9 points). The improvements cover all nine subtasks. Together with our original evaluations on Qwen2.5-VL and Qwen3-VL, this comparison provides further evidence that our method continues to deliver substantial gains as backbone capabilities improve.

In addition, we provide OVO-Bench results with LLaVA-OneVision-7B to further examine the applicability of our method to a different MLLM architecture. The results alongside ReKV are summarized below.

MethodRT Avg.BW Avg.Overall
LLaVA-OneVision-7B61.646.454.0
+ ReKV61.744.753.2
+ LatentStream (Ours)68.151.960.0

As seen, our LatentStream improves overall accuracy by 6.0 points over the LLaVA-OneVision-7B. These results complement the stronger-backbone evaluation and further support the applicability of our training-free framework across different MLLM architectures.

We incorporate the above experimental results and the corresponding discussion into Appendix Sec. E and Table 15 of the revised manuscript.


Finally, we would like to sincerely thank you for your careful review and valuable comments, which really help us to improve this work. We hope we addressed your concerns. Please let us know if you'd like any further information.

Reviewer_DyVQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentStream, a training-free framework for online streaming video understanding that progressively internalizes retrieved historical visual evidence into Latent Memory Tokens (LMTs). It combines Hierarchical Streaming Memory (HSM) for bounded memory management, Hierarchical Latent Memory Evolution (HME) for iterative evidence retrieval, and Progressive Confidence-guided Latent Memory Optimization (PMO) for test-time adaptation.

I find entropy-based unsupervised test-time optimization particularly interesting. PMO updates LMTs without ground-truth labels or MLLM fine-tuning. However, minimizing predictive entropy does not necessarily ensure answer correctness or faithful preservation of historical information.

Experiments on OVO-Bench, StreamingBench, and three offline long-video benchmarks demonstrate improved Video QA accuracy. Notably, LatentStream improves OVO-Bench accuracy from 54.0% to 64.2% with Qwen2.5-VL-7B, while Table 5 supports the effectiveness of PMO.

Overall, the results support the proposed approach's effectiveness, but historical information retention, advantages over conventional streaming Video-RAG, and actual streaming processing throughput remain insufficiently validated.

Critical strengths

Interesting training-free test-time optimization for streaming video understanding.\textbf{Interesting training-free test-time optimization for streaming video understanding.}

The paper proposes an interesting approach to progressively updating Latent Memory Tokens (LMTs) through entropy-based unsupervised test-time optimization, without requiring ground-truth labels or fine-tuning the underlying MLLM. In particular, the hierarchical confidence-guided optimization mechanism offers an alternative to conventional retrieval-based memory mechanisms. Table 5 provides empirical evidence that the proposed optimization objective improves downstream Video QA accuracy compared with no optimization and absolute-confidence optimization.

Critical weaknesses

1. Insufficient evidence that iterative latent memory evolution preserves historical information.\textbf{1. Insufficient evidence that iterative latent memory evolution preserves historical information.}

The paper claims that retrieved historical information is progressively internalized into LMTs. However, it does not explicitly establish a mechanism that prevents previously acquired information from being overwritten or lost during iterative LMT updates. Moreover, the experiments do not directly evaluate whether historical information is preserved throughout the optimization process.

This is particularly important because the proposed method is intended for long-horizon streaming video understanding, where retaining sparse but potentially important historical evidence is essential. The reported improvements in downstream Video QA accuracy do not necessarily demonstrate that the internalized information is faithfully preserved.

I recommend evaluating the proposed method using a Needle-in-a-Haystack (NIAH) experiment, varying the temporal position and retention duration of relevant information. An analysis of information retention across successive LMT optimization iterations would also help substantiate the central claim of progressive memory internalization.

2. Insufficient evaluation of streaming execution efficiency.\textbf{2. Insufficient evaluation of streaming execution efficiency.}

Although the paper does not explicitly claim to provide a hard real-time streaming system, it focuses on online MLLMs and emphasizes efficiency in streaming video understanding. Therefore, execution speed is an important factor in assessing its practical relevance.

The reported 1 fps specifies the video sampling rate rather than the actual processing throughput. While Table 7 reports Time to First Token (TTFT), Time per Output Token (TPOT), and peak GPU memory usage, these metrics do not establish whether the system can continuously process incoming video frames at the required rate.

In particular, it remains unclear whether video ingestion and hierarchical memory updates can continue without accumulating delays while the model performs iterative test-time optimization for question answering.

I recommend reporting the sustained streaming processing throughput (fps), per-frame processing latency, and end-to-end QA latency. Evaluating concurrent video ingestion and question answering would provide stronger evidence of the method's practical applicability to online streaming scenarios.

Questions for authors

1. Generalizability of the training-free approach\textbf{1. Generalizability of the training-free approach}

The paper emphasizes training-free adaptation as an important advantage, but the experiments are limited to the Qwen2.5-VL and Qwen3-VL model families.

To what extent does the proposed latent memory optimization depend on the specific architecture or internal representations of these models? Can the authors provide evidence that the approach generalizes to a substantially different MLLM architecture?

2. Evaluation metrics and sample-size imbalance\textbf{2. Evaluation metrics and sample-size imbalance}

Table 1 reports macro-averaged accuracy across sub-tasks and categories, potentially assigning equal importance to tasks with different numbers of evaluation samples.

Could the authors report the number of evaluation examples in each sub-task and provide sample-weighted micro-averaged accuracy in addition to the existing macro-averaged results? In particular, do the reported improvements over competing methods remain consistent under micro-averaged evaluation?

3. Practical advantages of training-free versus training-based approaches\textbf{3. Practical advantages of training-free versus training-based approaches}

Training-free adaptation is a meaningful design advantage. However, it is not necessarily a prerequisite for practical online video understanding, where training-based approaches may offer comparable or superior accuracy and inference efficiency.

For example, ThinkStream (3B), a training-based approach, achieves slightly higher overall accuracy than LatentStream (3B) on OVO-Bench (59.7% versus 59.0%).

Under what practical conditions do the authors expect the proposed training-free approach to be preferable to training-based online MLLMs? Can the authors provide a more explicit comparison of inference latency, memory usage, or overall computational costs, including training and inference?

4. Relationship to streaming Video-RAG methods\textbf{4. Relationship to streaming Video-RAG methods}

The proposed retrieve-and-internalize framework appears closely related to retrieval-augmented video understanding, with the additional mechanism of iterative latent memory optimization.

Although the paper compares against OASIS, which incorporates hierarchical memory retrieval, the distinction between the proposed approach and the broader family of streaming Video-RAG methods could be clarified.

What is the specific advantage of progressively evolving LMTs over conventional retrieval-based approaches under comparable memory and inference-computation budgets? Could the authors provide a more detailed discussion or a controlled comparison that isolates the contribution of latent memory internalization from iterative evidence retrieval?

5. Comparison with recent proprietary MLLMs\textbf{5. Comparison with recent proprietary MLLMs}

Table 1 includes Gemini 1.5 Pro and GPT-4o as proprietary MLLM baselines, both introduced in 2024.

Given the ICLR 2027 submission timeline, these baselines may not adequately reflect the capabilities of more recent proprietary multimodal models.

Could the authors update the comparison using newer models, or explain why such evaluations are infeasible? Any additional comparisons should clearly specify the video-input protocol and its compatibility with the online streaming setting.

8 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer DyVQ [Part 1]

We sincerely thank Reviewer DyVQ for the constructive feedback. We provide point-to-point responses below, and all corresponding revisions have been incorporated into the revised paper and highlighted in red. All references (e.g., Tables, Sections) correspond to the updated version.


W1: More evidence that iterative latent memory evolution preserves historical information.

A: Thank you for this helpful suggestion. To directly examine whether historical information is preserved throughout optimization, we add two complementary evaluations: a needle-in-a-haystack evaluation and a ablation study of historical information retention across successive updates of Latent Memory Tokens (LMTs).

(1) Needle-in-a-haystack evaluation. Following your suggestion, we evaluate Qwen2.5-VL-7B on the complete standard VNBench test set [ref1], comprising 1,350 samples across Retrieval, Ordering, and Counting, with 450 samples per category. VNBench inserts one-second subtitle or image needles at different temporal positions within background videos ranging from approximately 10 to 180 seconds. Questions concern the content, temporal order, or occurrence count of these needles. We process each video at 1 fps and answer the question after the video ends. Under the same evaluation setting, we compare three configurations. Qwen2.5-VL-7B answers directly from the sampled video frames. HSM-only constructs Hierarchical Streaming Memory (HSM) and answers directly from this memory without LMTs. Our LatentStream uses the same HSM and further performs iterative evidence retrieval and confidence-guided LMT optimization. The results are summarized below.

MethodRetrieval Avg.Ordering Avg.Counting Avg.Overall
Qwen2.5-VL-7B74.460.935.156.8
+ HSM-only75.862.037.658.4
+ LatentStream (Ours)78.765.844.763.0

As shown above, LatentStream improves overall accuracy over Qwen2.5-VL-7B and HSM-only by 6.2 and 4.6 points, respectively. Since the questions specifically concern the inserted needles, improvements across Retrieval, Ordering, and Counting provide targeted evidence of more effective recovery of sparse historical content, temporal order, and occurrence counts. The advantage over HSM-only further suggests that latent memory evolution helps the model make more effective use of retained historical evidence.

(2) Historical Information retention across successive LMT updates. To directly examine whether previously acquired information is overwritten during subsequent optimization, we conduct a ablation study on the Retrieval subset. For each sample, we construct an Hierarchical Streaming Memory (HSM) containing the target needle and a background Hierarchical Streaming Memory (HSM) excluding the target needle. Iterations 1 and 2 optimize LMTs using the HSM containing the target evidence. Iterations 3 and 4 use the fixed background HSM while carrying forward the LMT state from the preceding iteration. At every iteration, answers are generated using only the background HSM, the question, and the current LMTs. We report answer accuracy and continuous retention, defined as the percentage of samples with correct answers at iteration 2 that remain correct at every subsequent iteration. Continuous retention at iteration 2 is 100% by definition.

Optimization IterationHSM Used for OptimizationAnswer Accuracy (%)Continuous Retention (%)
1HSM containing the target needle76.4—
2HSM containing the target needle77.3100.0
3Background HSM excluding the target needle76.999.4
4Background HSM excluding the target needle76.799.1

As seen, at iteration 4, 99.1% of the samples with correct answers at iteration 2 remain correct throughout both subsequent updates, and answer accuracy decreases by only 0.6 points. These results provide direct empirical support that previously acquired historical information remains usable through successive LMT updates after its source visual evidence is removed.

We include these additional experimental results and the corresponding discussion in Appendix Sec. D.8, Table 12, and Table 13.

[ref1] Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs. ICLR 2025.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer DyVQ [Part 2]

W2: More evaluation results on streaming execution efficiency.

A: Thank you for your suggestion. To further examine streaming execution efficiency, we extend the analysis in Appendix Table 7 with three additional metrics: streaming processing throughput, per-frame processing latency, and end-to-end QA latency. We compare LatentStream with Qwen2.5-VL-7B under the same OVO-Bench evaluation settings and hardware configuration.

We first clarify the timing protocols. The original TTFT in Table 7 includes memory construction. In contrast, the additional end-to-end QA latency measures the time from query arrival to completion of answer generation. Streaming processing throughput measures the number of sampled frames fully processed per second. Per-frame processing latency measures the time from the arrival of a sampled frame to completion of its visual processing and any associated memory update. The results are summarized below.

Methodstreaming processing throughput ↑ (fps)Per-frame processing latency ↓ (ms)End-to-end QA latency ↓ (s)
Qwen2.5-VL-7B25FPS40ms2.64s
+ LatentStream (Ours)22FPS45ms2.78S

As shown above, our LatentStream achieves 22 fps processing throughput and 45 ms per-frame processing latency. Its frame-processing throughput exceeds the 1 fps sampling rate used in our evaluation, indicating sufficient processing capacity for visual encoding and online HSM updates in the measured frame-processing pipeline. Meanwhile, end-to-end QA latency increases from 2.64s to 2.78s, introducing only 0.14s of additional latency over the backbone. The compact HSM shortens the visual context processed at query time, which can partially offset the additional computation required by iterative latent memory optimization. These results show that LatentStream maintains efficient frame processing with a modest increase in query-time latency under the tested configuration.

We clarify the timing protocols and include these additional results and the corresponding discussion in Appendix D.1 and Table 7.

Q1: The generalizability of our training-free method.

A: Good comment! We would like to clarify that our latent memory optimization operates in each backbone’s native embedding space and uses its predictive distributions as feedback, without requiring Qwen-specific architectural components. The Latent Memory Tokens (LMTs) are instantiated with the corresponding embedding dimension, while the confidence-based objective and LMT update rule remain unchanged. Both the visual encoder and language model are kept frozen throughout inference.

Following your suggestion, we additionally conduct additional experiments with LLaVA-OneVision-7B on OVO-Bench and StreamingBench to examine the applicability of our method beyond the Qwen-VL architectures. The complete task-wise results are presented below.

Results on OVO-Bench with LLaVA-OneVision-7B.

MethodOCRACRATRSTUFPDOJRRT Avg.EPMASIHLDBW Avg.Overall
LLaVA-OneVision-7B58.459.669.844.476.261.461.653.954.730.646.454.0
+ ReKV [ref2]63.155.172.446.672.360.961.755.263.415.544.753.2
+ LatentStream (Ours)71.162.476.757.975.265.268.157.664.233.951.960.0

Results on StreamingBench with LLaVA-OneVision-7B.

MethodOPCRCSATPEUTRPRSUACPCTAll
LLaVA-OneVision-7B76.877.282.675.364.264.275.061.461.246.168.4
+ ReKV65.978.177.971.866.965.169.462.261.143.166.2
+ LatentStream (Ours)82.682.079.283.078.973.873.173.268.052.875.1

As shown above, our LatentStream improves upon the LLaVA-OneVision-7B backbone by 6.0 and 6.7 points on OVO-Bench and StreamingBench, respectively, and exceeds ReKV results. Together with the original Qwen2.5-VL and Qwen3-VL results, these experiments further support the generalizability of our training-free approach across different MLLM architectures.

We incorporate the above experimental results and the corresponding discussion into Section 4.2, with the complete OVO-Bench and StreamingBench results reported in Tables 1, 15 and 16, respectively.

[ref2] Streaming video question-answering with in-context video kv-cache retrieval. ICLR 2025.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer DyVQ [Part 3]

Q2: Evaluation metrics and sample-size imbalance.

A: Thank you for this helpful suggestion. Table 1 follows the benchmark’s evaluation protocol, averaging sub-task accuracies within Real-Time Visual Perception (RT) and Backward Tracing (BW) before averaging the two category scores. Following your suggestion, we additionally report the number of evaluation examples in each sub-task and provide sample-weighted micro-averaged accuracy. The evaluation covers 1,468 examples across nine sub-tasks, with the following distribution:

CategorySub-taskNumber of examples
RTOCR149
RTACR109
RTATR116
RTSTU178
RTFPD101
RTOJR184
RT subtotal837
BWEPM297
BWASI148
BWHLD186
BW subtotal631
Total9 sub-tasks1,468

We further present sample-weighted micro-averaged accuracy for the training-free online methods in Table 1 below. Specifically, we weight each sub-task accuracy by its number of evaluation examples and aggregate across all 1,468 examples, giving each example equal weight. We present the training-free online methods here for brevity. We provide the complete sample-weighted micro-averaged results and the sub-task sample counts and corresponding discussion in Appendix Sec. G and Table 18.

BackboneMethodMacro-averaged accuracy (%)Sample-weighted micro-averaged accuracy (%)
Qwen2.5-VL-3BBackbone52.253.2
+ FluxMem53.354.6
+ LatentStream (Ours)59.059.3
Qwen2.5-VL-7BBackbone54.055.0
+ FluxMem56.957.7
+ OASIS60.061.0
+ LatentStream (Ours)64.263.3

As shown above, the improvements over competing methods remain consistent under micro-averaged evaluation. These results show that our comparative advantages hold under both task-balanced macro averaging and sample-weighted micro averaging.

Q3: Practical advantages of training-free versus training-based approaches.

A: To solidly address your concern, we clarify (i) the practical conditions that favor training-free streaming video method and provide (ii) an explicit comparison of training and inference costs between LatentStream-3B (training-free method) and ThinkStream-3B (training-based method) below.

(1) Practical conditions. Our LatentStream enhances pretrained backbones for streaming video understanding without additional task-specific training data or backbone retraining. This is particularly useful when (i) training data and computational resources for additional adaptation are limited, as it avoids collecting streaming-specific supervision and performing additional model training; or (ii) backbone models are frequently updated, as our memory construction and latent optimization mechanisms can be reused with compatible backbones, reducing repeated training costs.

(2) Training requirements and inference costs. Following your suggestion, we further compare LatentStream-3B with ThinkStream-3B on OVO-Bench, covering both additional training requirements and inference costs. ThinkStream requires 120 H20 GPU hours for additional training on 110K Cold Start examples and 9K examples for Reinforcement Learning with Verifiable Rewards (RLVR). In contrast, LatentStream requires neither additional training data nor backbone retraining. The inference results under the same evaluation settings are summarized below. Per-frame processing latency measures the time from frame arrival to completion of the corresponding online state update. End-to-end QA latency measures the time from question arrival to complete answer generation.

MethodAdditional TrainingAdditional Training DataAdditional Training Cost (H20 GPU-hours)OVO-Bench (%)Per-frame Latency (ms)QA Latency (s)Peak GPU Memory (GB)
ThinkStream-3BYes119K12059.73501.414.2
LatentStream-3BNoNone059.0451.613.5

As shown above, LatentStream reduces per-frame processing latency from 350 to 45 ms and peak GPU memory from 14.2 to 13.5 GB, while incurring 0.2 s of additional QA latency. It also avoids the additional training data and 120 H20 GPU hours required by ThinkStream. These results support LatentStream as a practical option when resources for additional training are limited or compatible backbones are frequently updated, with modest trade-offs in accuracy and QA latency.

We have incorporated these comparisons and the corresponding discussion of practical conditions and computational trade-offs into Appendix Sec. D.2 and Table 8 of the revised manuscript.

Reviewer_ThSROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LatentStream, a training-free framework that improves streaming video QA. It first constructs short, mid, and long-term visual memories through Jenks-based thresholding. When a query arrives, it initializes three groups of latent memory tokens with different retrieval scopes, iteratively retrieves historical visual evidence, and updates both the latent tokens and temporary visual representations through policy-gradient guided by predictive entropy. Finally, the temporary evidence is removed, and the answer is generated using the external memory, the query, and the optimized latent tokens. LatentStream demonstrates improvements across all benchmarks.

Critical strengths
  • LatentStream improves performance across multiple benchmarks and demonstrates similar gains across additional backbones on streaming benchmarks, supporting its transferability.
  • The method is described very clearly.
  • The ablation studies are reasonably comprehensive and cover the key components.
Critical weaknesses
  • The proposed method is evaluated on offline benchmarks only with Qwen2.5-VL. Evaluation with newer models, such as Qwen3-VL or Qwen3.5, would help distinguish whether the gains observed on newer backbones primarily come from adapting offline models to streaming evaluation or reflect broader improvements in video understanding.
  • These newer models also substantially extend the range of context lengths over which they maintain strong performance, from 32K to 256K tokens. The method’s effectiveness with video inputs at these longer context lengths requires further investigation.
  • The memory budget is currently fixed at 2,048 tokens. The effect of this choice should be explored further.
Questions for authors
  • Could the authors evaluate the method on offline benchmarks using newer backbones and compare it against recent offline baselines?
  • Could the authors select a backbone supporting longer contexts and evaluate different memory budgets? For example, after sampling an entire long video at 1 fps and a relatively high resolution, would the method remain effective when compressing several hundred thousand input tokens into a 32K-token memory?

I am willing to raise my score if these concerns are addressed.

4 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer ThSR [Part 1]

We sincerely thank Reviewer ThSR for the constructive feedback. We provide point-to-point responses below, and all corresponding revisions have been incorporated into the revised paper and highlighted in red. All references (e.g., Tables, Sections) correspond to the updated version.


W1&Q1: Offline evaluation with newer backbones.

A: Thank you for this valuable suggestion. Our original submission shows that our LatentStream improves Qwen3-VL-8B on streaming benchmarks, increasing the StreamingBench from 73.2 to 82.0 and the OVO-Bench from 61.8 to 73.8 (Fig. 5(a) and Tables 15-16).

Following your suggestion, we provide additional offline benchmark evaluation results below with Qwen3-VL-8B on three offline benchmarks: VideoMME, MLVU, and LongVideoBench. We have also expanded the comparison to include two recent training-free offline methods, AdaQ [ref2] and QCA [ref3], alongside the backbone and the recent streaming method OASIS [ref1]. For AdaQ and QCA, we quote the results from their original papers, which use 64 keyframes selected from candidate frames sampled at 1 fps. Under the same evaluation setting, our LatentStream improves upon the backbone Qwen3-VL-8B on VideoMME, MLVU, and LongVideoBench by 2.2, 2.1, and 3.7 points, respectively, and also outperforms OASIS, AdaQ and QCA. These offline results with Qwen3-VL-8B suggest that LatentStream’s gains extend beyond adapting offline models to streaming evaluation and reflect broader improvements in video understanding.

MethodVideoMMEMLVULongVideoBench
Qwen3-VL-8B71.776.963.7
+ AdaQ [ref2]69.676.466.9
+ QCA [ref3]69.575.766.9
+ OASIS [ref1]72.578.165.8
+ LatentStream (Ours)73.979.067.4

We add these results and the corresponding discussion to the Appendix Sec. F and Table 17 of the revised manuscript.

[ref1] OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning. CVPR 2026.

[ref2] Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling. Arxiv, 2026.

[ref3] QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding. ECCV 2026.

W2&Q2: Effectiveness under long-context settings.

A: Following your suggestion, we conduct additional long-input experiments with Qwen3-VL-8B on VideoMME and OVO-Bench. Specifically, we sample each full-length video on VideoMME and only the observed video prefix up to each query timestamp on OVO-Bench, at 1 fps with 128 visual tokens per frame, yielding inputs of up to 256K tokens. These inputs are progressively compressed into a Hierarchical Streaming Memory (HSM). We evaluate our LatentStream with both the original 2K memory budget and an enlarged 32K budget.

For a fair comparison, we evaluate our LatentStream against two Qwen3-VL-8B baselines: (i) a uniform-sampling baseline, which uniformly selects frames from the same video stream within a 32K visual-token budget, retaining 128 visual tokens per selected frame; and (ii) a full-input baseline, which directly processes the same uncompressed input of up to 256K tokens. The comparison results are provided below. Note that the slight differences between the 2K-budget results reported here and those in the original evaluation reflect the updated setting: complete videos on VideoMME and complete observed prefixes on OVO-Bench, with 128 visual tokens per frame, whereas the original online/offline evaluations used at most 256/1,024 frames.

MethodVisual Token BudgetVideoMMEOVO-Bench
Qwen3-VL-8B (uniform sampling)32K67.360.8
Qwen3-VL-8B (full input)Up to 256K71.761.8
+ LatentStream (Ours)2K74.173.7
+ LatentStream (Ours)32K75.874.9

As seen, our LatentStream outperforms both baselines on both benchmarks with either memory budget. Increasing the budget from 2K to 32K further improves accuracy by 1.7 and 1.2 points. These results demonstrate that our LatentStream remains effective under long-context settings, even when the backbone directly processes up to 256K visual tokens, and benefits further from a larger memory budget.

The above results and the corresponding discussion are incorporated into the Appendix Sec. D.5 and Table 10 of the revised manuscript.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer ThSR [Part 2]

W3&Q2: The impact of different memory budgets.

A: Good comment! Following your suggestion, we have conducted additional experiments to examine the effect of the Hierarchical Streaming Memory (HSM) budget using Qwen3-VL-8B on VideoMME and OVO-Bench. We vary the total memory budget over 2K, 4K, 8K, 16K, and 32K visual tokens, while keeping the same long-video inputs and other settings unchanged. All videos are sampled at 1 fps with 128 visual tokens per frame. All methods use the same available video segment, with no access to future frames on OVO-Bench. For comparison, we also include the backbone directly processing all visual tokens, with inputs of up to 256K tokens. The results are summarized below.

MethodVisual Token BudgetVideoMMEOVO-Bench
Qwen3-VL-8B (full input)Up to 256K71.761.8
+ LatentStream (Ours)2K74.173.7
+ LatentStream (Ours)4K74.974.3
+ LatentStream (Ours)8K75.574.6
+ LatentStream (Ours)16K75.774.8
+ LatentStream (Ours)32K75.874.9

As seen, LatentStream consistently improves as the memory budget increases, with diminishing gains at larger budgets. Even with a 2K budget, it outperforms full-input processing by 2.4 and 11.9 points on VideoMME and OVO-Bench, respectively. These results demonstrate that LatentStream remains effective across different memory budgets and can benefit from additional memory capacity.

We retain 2K as the default budget for two practical reasons: (i) it matches the memory budget of FluxMem, ensuring a fair comparison; and (ii) it provides lower inference latency than larger budgets while already achieving strong accuracy on both benchmarks. The above discussion and statistics are added in Appendix Sec. D.6 and Table 10.


Finally, we would like to sincerely thank you for your careful review and valuable comments. We have revised our manuscript with discussions and experimental results according to your review. It is welcome if there are any further questions or suggestions.

Thanks for your reply. My concerns have been addressed. I will raise my score to 3.

AuthorsOpenReview ↗

Response to Reviewer ThSR

Dear Reviewer ThSR,

Thank you for your acknowledgment of our work and for the efforts you put into this review process. If you have any further questions or suggestions, please feel free to share them.

Sincerely yours,

Authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

To all reviewers:

We express our sincere gratitude to all reviewers for their valuable time and thorough assessment of our manuscript. We appreciate the positive feedback on our training-free latent memory optimization (Reviewers DyVQ and J19j), the empirical improvements across online and offline benchmarks (Reviewers ThSR and J19j), and the clarity of presentation (Reviewers ThSR and J19j).

We have provided point-by-point responses and revised the manuscript accordingly, with the corresponding changes highlighted in red. All section and table references below refer to the revised version. The major changes are as follows:

  1. We have added offline evaluations with Qwen3-VL-8B on VideoMME, MLVU, and LongVideoBench, following Reviewer ThSR’s suggestions (Appendix Sec. F and Table 17).

  2. We have added long-context evaluations with inputs of up to 256K visual tokens. We have also examined Hierarchical Streaming Memory (HSM) budgets ranging from 2K to 32K, following Reviewer ThSR’s comments (Appendix Secs. D.5–D.6 and Table 10).

  3. We have expanded the comparisons to include SimpleStream and SAVEMem and added an experiment applying our latent memory evolution strategy to SimpleStream under the same eight-frame input, following Reviewer J19j’s suggestions (Appendix Secs. D.7 and E, and Tables 11 and 15–16).

  4. We have revised the OVO-Bench discussion to highlight the episodic memory and action sequence identification improvements on Qwen3-VL-8B, following Reviewer J19j’s comments (Sec. 4.2).

  5. We have added an ablation study to examine whether historical information remains usable through the Latent Memory Tokens (LMTs), following Reviewer J19j’s comments (Sec. 4.3 and Table 4).

  6. We have added a needle-in-a-haystack evaluation on VNBench and an analysis of historical information retention across successive LMT updates, following Reviewer DyVQ’s suggestions (Appendix Sec. D.8 and Tables 12–13).

  7. We have clarified the relationship between LatentStream and streaming Video-RAG methods and added controlled comparisons to examine the individual and complementary contributions of iterative evidence retrieval and latent memory optimization, following Reviewer DyVQ’s comments (Appendix Sec. D.9 and Table 14).

  8. We have added evaluations with Qwen3.5-9B and LLaVA-OneVision-7B to examine whether the benefits persist with a stronger backbone and generalize across different MLLM architectures, following comments from Reviewers J19j and DyVQ (Sec. 4.2, Appendix Sec. E, and Tables 15–16).

  9. We have extended the efficiency analysis to report streaming processing throughput, per-frame processing latency, and end-to-end QA latency, following Reviewer DyVQ’s suggestions (Appendix Sec. D.1 and Table 7).

  10. We have clarified the practical conditions favoring training-free adaptation and added a comparison with ThinkStream-3B covering additional training data and computational costs, inference latency, and peak GPU memory, following Reviewer DyVQ’s comments (Appendix Sec. D.2 and Table 8).

  11. We have reported the evaluation sample counts for individual OVO-Bench subtasks and added sample-weighted micro-averaged accuracy, following Reviewer DyVQ’s suggestions (Appendix Sec. G and Table 18).

  12. We have updated the proprietary MLLM comparisons to include GPT-5.1 and Gemini 3 Flash and clarified their video-input protocols, following Reviewer DyVQ’s suggestions (Table 1).

For further details, please refer to our responses to each reviewer. We sincerely appreciate your constructive feedback and welcome further questions and suggestions.

Sincerely yours,

Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.