- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LatentStream, a training-free framework for streaming video understanding. It maintains a hierarchical visual memory and uses a small set of latent memory tokens to iteratively retrieve and internalize query-relevant historical evidence. A confidence-based objective optimizes these tokens at test time while keeping the backbone frozen. Experiments on online and offline video benchmarks show improvements over the base models.
Critical strengths
- The paper addresses a practical problem: improving streaming video question answering under a fixed memory budget without updating the backbone parameters.
- Combining historical evidence retrieval with test-time optimization of latent tokens is an interesting idea.
- The experiments cover both online and offline tasks and show performance gains across these settings.
Critical weaknesses
- Missing key baselines. The paper does not compare against SimpleStream [1] or SAVEMem [2], the latter already cited in the paper. A comparison with a simple recent-frame window under matched settings would help establish whether the additional retrieval and optimization mechanisms are necessary.
- Limited evidence of broader memory improvements. On Qwen2.5-VL-7B, the gain in Backward Tracing comes almost entirely from hallucination detection, with little improvement in episodic memory or action sequence identification (Table 1). The higher average score therefore does not establish a broad improvement in long-term memory.
- Insufficient validation of the “internalization” mechanism. The model still accesses the original visual memory bank during final answer generation (Sec. 3.4). The ablations in Table 4 show that optimizing latent tokens helps, but do not distinguish whether these tokens store historical information or simply help the model read the existing memory more effectively.
[1] Yujiao Shen, Shulin Tian, Jingkang Yang, and Ziwei Liu. A Simple Baseline for Streaming Video Understanding. arXiv:2604.02317, 2026.
[2] Hang Wu, Sherin Mary Mathews, Yujun Cai, Ming-Hsuan Yang, and Yiwei Wang. Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding. arXiv:2605.07897, 2026.
Questions for authors
Would the method still provide substantial gains with a stronger backbone, such as Qwen3.5 or a larger model? It would be useful to understand whether the benefits persist as the backbone’s capabilities improve.
6 comments
Point-to-Point Response to Reviewer J19j [Part 2]
W2: Evidence of broader memory improvements.
A: Thank you for your thoughtful comment. We would like to clarify that, alongside the substantial HLD improvement (23.7→68.3), our LatentStream also improves ASI from 58.8 to 60.8 (+2.0 points) on Qwen2.5-VL-7B, indicating better performance in identifying historical action sequences. EPM shows a small decrease from 51.5 to 50.8. Thus, while HLD contributes the largest gain on this backbone, the benefits also extend to action sequence identification, with the magnitude of improvement varying across memory subtasks.
Our original submission further reports improvements in both EPM and ASI on Qwen3-VL-8B in Appendix Table 15. For clarity, we reproduce the relevant results below.
| Method | EPM | ASI | HLD | BW Avg. |
|---|---|---|---|---|
| Qwen3-VL-8B | 52.5 | 60.1 | 46.8 | 53.1 |
| + OASIS | 62.0 | 60.1 | 47.3 | 57.2 |
| + LatentStream (Ours) | 64.3 | 64.9 | 72.6 | 67.3 |
As seen, LatentStream improves EPM and ASI over the backbone by 11.8 and 4.8 points, respectively, and outperforms OASIS by 2.3 and 4.8 points. Even when HLD is excluded, the mean accuracy across EPM and ASI increases from 56.3 to 64.6 (+8.3 points). These task-specific results provide direct evidence of a broad improvement in long-term memory on Qwen3-VL-8B.
The offline evaluations provide complementary evidence of improved long-video understanding. As reported in Table 2, LatentStream improves Qwen2.5-VL-7B by 3.3, 6.1, and 1.4 points on VideoMME, MLVU, and LongVideoBench, respectively.
We revise the OVO-Bench discussion in Section 4.2 to explicitly highlight the EPM and ASI improvements on Qwen3-VL-8B and their support for broader memory benefits.
W3: Validation of the internalization mechanism.
A: Thank you for pointing out this important distinction. Table 4 demonstrates the benefit of optimizing Latent Memory Tokens (LMTs). To directly examine whether the optimized Latent Memory Tokens (LMTs) store historical information or primarily help the model use the existing visual memory, we conduct an additional ablation below by removing the entire Hierarchical Streaming Memory (HSM) after LMT optimization. Specifically, we keep the optimization procedure unchanged, while generating the final answer using only the query and evolved LMTs. This setting tests whether information acquired during optimization remains usable through the LMTs after the original visual memory is removed.
| Method | OVO-Bench | VideoMME |
|---|---|---|
| HSM Only | 58.1 | 65.1 |
| HSM + Retrieved Visual Evidence | 59.7 | 65.6 |
| HSM + Initial Latent Memory Tokens | 56.5 | 65.2 |
| Evolved Latent Memory Tokens Only (w/o HSM) | 63.6 | 66.2 |
| HSM + Evolved Latent Memory Tokens (Ours) | 64.2 | 66.6 |
As seen, the evolved-LMT-only variant achieves 63.6% and 66.2% on OVO-Bench and VideoMME, respectively, outperforming HSM + Retrieved Visual Evidence by 3.9 and 0.6 points. Since the final decoder has no access to HSM in this setting, these results support the conclusion that the evolved LMTs store historical information relevant to the current query, rather than merely facilitating access to the existing memory. Furthermore, the full method improves over the LMT-only variant by 0.6 and 0.4 points, respectively, supporting the complementary contributions of evolved LMTs and HSM. These results suggest that LMTs serve as compact, query-conditioned working memory that retains historical information, while access to HSM provides additional visual context for answering.
We have incorporated this additional ablation into Table 4 and clarified the role of LMTs and their relationship with HSM in Sec. 4.3 of the revised manuscript.
Point-to-Point Response to Reviewer J19j [Part 3]
Q1: Consistent gains with stronger backbones.
A: Good comment! Following your suggestion, we conduct additional experiments on OVO-Bench with Qwen3.5-9B to examine whether our LatentStream continues to provide substantial gains as backbone capabilities improve. Both variants use the same frame sampling and visual input budget.The task-wise results are as follows.
| Method | OCR | ACR | ATR | STU | FPD | OJR | RT Avg. | EPM | ASI | HLD | BW Avg. | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-VL-9B | 83.9 | 62.4 | 70.7 | 46.1 | 76.2 | 70.6 | 68.3 | 61.6 | 62.8 | 59.1 | 61.2 | 64.8 |
| + LatentStream (Ours) | 95.3 | 83.5 | 84.5 | 73.0 | 77.2 | 83.2 | 82.8 | 65.7 | 66.2 | 74.2 | 68.7 | 75.7 |
As shown above, our LatentStream improves overall accuracy from 64.8% to 75.7% (+10.9 points). The improvements cover all nine subtasks. Together with our original evaluations on Qwen2.5-VL and Qwen3-VL, this comparison provides further evidence that our method continues to deliver substantial gains as backbone capabilities improve.
In addition, we provide OVO-Bench results with LLaVA-OneVision-7B to further examine the applicability of our method to a different MLLM architecture. The results alongside ReKV are summarized below.
| Method | RT Avg. | BW Avg. | Overall |
|---|---|---|---|
| LLaVA-OneVision-7B | 61.6 | 46.4 | 54.0 |
| + ReKV | 61.7 | 44.7 | 53.2 |
| + LatentStream (Ours) | 68.1 | 51.9 | 60.0 |
As seen, our LatentStream improves overall accuracy by 6.0 points over the LLaVA-OneVision-7B. These results complement the stronger-backbone evaluation and further support the applicability of our training-free framework across different MLLM architectures.
We incorporate the above experimental results and the corresponding discussion into Appendix Sec. E and Table 15 of the revised manuscript.
Finally, we would like to sincerely thank you for your careful review and valuable comments, which really help us to improve this work. We hope we addressed your concerns. Please let us know if you'd like any further information.
Point-to-Point Response to Reviewer J19j [Part 1]
We sincerely thank Reviewer J19j for the insightful comments. We address each point in detail below, and all corresponding revisions have been incorporated into the updated manuscript and highlighted in red. All references (e.g., Tables, Sections) refer to the revised version.
W1: Comparison with Key Baselines.
A: Thank you for pointing out these two relevant works. To solidly address your concern, we add comparisons with SimpleStream [ref1] (using the most recent eight frames) and SAVEMem [ref2] and conduct an additional experiment to examine whether our latent memory evolution strategy improves a simple recent-frame baseline.
(1) Comparison with SimpleStream and SAVEMem. The table below summarizes the published results of these baselines alongside the results from our original submission. For brevity, we report category-level and overall accuracy here; complete task-wise results are included in the revised manuscript.
As seen, with Qwen2.5-VL-7B, our LatentStream outperforms SimpleStream and SAVEMem by 0.5 and 1.5 points on OVO-Bench overall accuracy, respectively, with larger gains of 9.2 and 9.6 points on Backward Tracing (BW). With Qwen3-VL-8B, our LatentStream improves upon SimpleStream by 6.4 points on OVO-Bench overall, 12.4 points on OVO-Bench BW, and 3.2 points on StreamingBench. These results support the benefit of our approach, particularly for questions requiring historical information.
We also acknowledge that SimpleStream achieves better real-time perception (RT) and StreamingBench performance with Qwen2.5-VL-7B. We hypothesize that this difference may reflect how the two methods organize visual context. SimpleStream uses only the most recent eight frames, concentrating on recent visual details. In contrast, our LatentStream additionally incorporates historical memory and optimized latent memory to support reasoning. For questions primarily concerning the current scene, this additional context may divert attention from critical visual cues.
(2) Applying our latent memory evolution strategy to SimpleStream. To directly examine its contribution under the same recent-frame input, we incorporate our strategy into SimpleStream with Qwen3-VL-8B. Specifically, we retain the same eight-frame window and use its visual tokens as a fixed evidence pool. At query time, we iteratively retrieve relevant evidence from this pool to optimize a small set of Latent Memory Tokens (LMTs) through confidence-guided test-time optimization. Both variants access exactly the same eight frames, with no access to observations outside this window. The results under the same evaluation settings are summarized below.
As seen, incorporating our latent memory evolution strategy improves SimpleStream by 3.0 and 2.1 points on OVO-Bench and StreamingBench, respectively. These results support the added value of our retrieval and latent memory optimization mechanisms beyond a simple recent-frame window and demonstrate that our strategy can also be extended to recent-frame window methods.
We have incorporated the baseline comparisons, the experiment applying our strategy to SimpleStream, and the corresponding discussions into Appendix Table 15, Table 11 and Sec. D.7 of the revised manuscript.
[ref1] A simple baseline for streaming video understanding. Arxiv 2026.
[ref2] Semantic-aware adaptive visual memory for streaming video understanding. Arxiv 2026.