- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DualStream, a training-free framework for long-horizon streaming video understanding that integrates compact visual memory with textual episodic memory through a closed-loop feedback mechanism. The key idea is to use an evolving episodic summary to generate a future focus that guides subsequent visual token selection, rather than relying solely on visual redundancy. The framework is question-agnostic and operates with frozen VLM backbones.
The authors evaluate DualStream on multiple long-video and streaming QA benchmarks. On Video-MME Long, it achieves 60.0% accuracy while discarding 86.1% of visual tokens, outperforming the strongest reported online baseline by 4.7 percentage points. Ablation studies demonstrate that episodic feedback improves accuracy over visual-only memory by 2.22 percentage points under the same token budget. Experiments with Qwen2.5-VL-7B and InternVL3-8B also demonstrate transferability without additional training.
The primary contribution is improving long-horizon video understanding under constrained visual memory budgets. However, the evaluation mainly establishes improvements in aggregate QA accuracy, rather than directly characterizing long-term information retention, task-specific capabilities, or practical advantages over retrieval-based memory architectures.
Critical strengths
Training-free framework.
The proposed framework is training-free and does not require updating the parameters of the underlying VLM. This is a practical advantage because it avoids additional training costs and allows the proposed episodic feedback mechanism to be transferred to different VLM backbones. The experiments with Qwen2.5-VL-7B and InternVL3-8B provide initial evidence of this transferability, although model-specific adaptation and calibration are still required.
Critical weaknesses
1. Limited experimental analysis of long-horizon understanding capabilities.
The evaluation primarily reports overall accuracy on several video understanding benchmarks. However, these aggregate results do not sufficiently clarify which specific capabilities benefit from the proposed episodic feedback mechanism and where its limitations lie. A more fine-grained evaluation across task subcategories, such as the analyses presented in QViC-MF (CVPR 2026), would be valuable. In particular, Needle-in-a-Haystack (NIAH) evaluations are important for assessing whether the proposed selective visual memory can preserve rare but potentially critical information over long time horizons. Without such analyses, the extent to which DualStream improves long-term information retention remains unclear.
2. Insufficient comparison with retrieval-based memory approaches.
One of the main practical motivations of DualStream appears to be accurate long-horizon streaming video understanding under constrained memory resources. However, the paper does not directly compare the proposed framework with relevant retrieval-based approaches, such as Video-RAG (NeurIPS 2025) and StreamRAG (CVPR 2026), or temporal graph-based memory approaches. Although these methods may differ in their assumptions and memory architectures, they address closely related challenges in efficiently retaining and accessing information from long video streams. Comparisons under clearly defined and comparable memory, latency, and accuracy constraints would help establish the practical advantages of DualStream. Without such comparisons, its engineering value relative to alternative memory architectures is not sufficiently demonstrated.
Questions for authors
1. Potential for model-agnostic generalization
One of the strengths of DualStream is its training-free design. However, Appendix D indicates that applying the framework to different VLM backbones requires architecture-specific modifications and hyperparameter calibration. Could the authors clarify whether the proposed episodic feedback mechanism could be extended to a model-agnostic framework without relying on backbone-specific attention extraction or token representations? Achieving such generality would substantially strengthen the novelty and practical applicability of the proposed approach.
2. Real-time streaming feasibility
The reported mean visual-state update latency is 625.95 ms on an NVIDIA A100, corresponding to approximately 1.6 fps for this component alone. While this supports the 1-fps setting, can DualStream sustain real-time processing when visual and episodic memory updates run concurrently? How does the effective throughput change under higher frame sampling rates or substantially longer streams?
3. End-to-end computational efficiency
Although DualStream discards 86.1% of visual tokens, the additional attention scoring and episodic memory updates introduce computational overhead. How does the total GPU memory consumption and end-to-end computational cost compare with the visual-only baseline under identical conditions? Does the improvement in QA accuracy justify the additional computational overhead?
4 comments
Response to Reviewer Qpyx (2/2)
Q1. Potential for model-agnostic generalization
The episodic feedback loop transfers across backbones: retained evidence updates episodic understanding, and the resulting focus guides subsequent visual retention. Our current implementation translates that focus into token scores through the backbone’s internal attention.
Appendix D in the original manuscript describes the adapters needed to map this shared selection rule onto each backbone’s visual-token layout, position encoding, and attention interface. These adaptations keep all model parameters frozen. Layer selection and compression calibration use unlabeled visual inputs and generic focus prompts.
These adaptations accommodate differences between backbones while preserving the same episodic feedback mechanism.
Q2. Real-time streaming feasibility
Our target setting is a 1-fps input stream, which is a typical setting used in streaming video understanding research. As reported in Appendix C.3 of the original submission, the mean duration of the complete visual-state update is 625.95 ms, and its p99 latency is 970 ms, both within the one-second input interval.
Episodic generation operates at a much lower frequency and accounts for 2.46% of video duration. Visual selection uses the most recently completed focus while a new episodic update is being generated. Thus, the visual stream has no dependency on the completion of the new focus. Together, these measurements support the average processing budget at 1 fps.
Sampling rates above our default of 1 fps require a larger computational budget and make processing backlogs more likely. Longer videos do not affect the throughput of visual updates, since these updates process only newly arrived visual tokens and the computation is independent of the elapsed video duration.
Q3. End-to-end computational efficiency
The main additional per-frame operation over Visual-only is attention scoring and selection, which takes 177.22 ms on average in the original submission’s runtime breakdown. Sparse episodic memory generation introduces a further cost, with cumulative generation time accounting for 2.46% of processed video duration. Under real-time input, elapsed processing time is largely determined by video duration when processing keeps pace with incoming frames. Per-frame updates use the available time between frame arrivals. Finishing a frame earlier within that interval does not increase ingestion throughput.
During rebuttal, we also measured GPU memory using the same profiling configuration as the original runtime analysis. Each measurement covers processing a complete video and answering its three questions, including model weights and resident memory. We report the mean and maximum across per-video peaks. All values are in GiB.
| Configuration | Peak allocated: mean | Peak allocated: max |
|---|---|---|
| Visual-only | 22.59 | 25.39 |
| DualStream | 22.86 | 25.66 |
DualStream increases both the mean and maximum allocated peaks by 0.27 GiB. The additional per-frame computation mainly involves attention scoring on newly arrived visual tokens. Both methods retain model weights and visual memory, which account for the bulk of allocated GPU memory.
These measurements indicate a modest increase in allocated GPU memory and additional computation from attention-based selection and episodic generation, alongside accuracy gains of 2.22 percentage points on Video-MME Long and 1.70 points on MLVU.
We thank the reviewer for these constructive suggestions. We will include the new experimental results and corresponding clarifications in the revision.
Thank you for the additional experiments and explanations addressing my questions. However, I remain concerned that the effectiveness of episodic feedback for long-horizon video understanding, a central motivation of DualStream, has not been sufficiently demonstrated.
In the task-level MLVU results, DualStream shows a relatively modest overall improvement over the Visual-only baseline (+1.70 percentage points), while the improvement on Needle QA is almost negligible (+0.28 percentage points). Although the gains in some other task categories are encouraging, these results do not clearly establish that episodic feedback improves long-term information retention, particularly for rare but potentially important information.
Thank you for your response.
As described in the original MLVU paper, Needle QA is constructed by randomly inserting a short clip from another video into a background video. Manual verification further ensures that the needle scene does not appear in the background video. This construction gives the inserted needle no established connection to the surrounding episode. In a query-agnostic streaming setting, guidance derived from the surrounding episode is therefore not expected to prioritize the needle.
The gains in counting, anomaly recognition, and action ordering demonstrate the benefits of episodic feedback for understanding events across a long video, where visually similar observations can have different meanings in the unfolding episode.
Thank you again for your consideration. We would appreciate your thoughts on this distinction between retaining independently inserted details and understanding events within an unfolding episode.
Response to Reviewer Qpyx (1/2)
We thank the reviewer for recognizing our training-free design and contribution. We address the two weaknesses and the three questions below.
W1. Capability analysis and long-term information retention
To examine performance across capabilities, we added task-level MLVU results for the Visual-only baseline during rebuttal and compared them with the DualStream and SimpleStream results from the original submission. Overall accuracy is computed over all evaluation questions.
The largest gains over Visual-only occur in counting, anomaly recognition, and action ordering. A possible explanation is that episodic memory explicitly records important events and their relationships for later question answering, while the generated focus guides subsequent visual selection toward semantically relevant evidence. In contrast, the visual-only baseline tends to filter out visually repetitive but semantically meaningful content.
MLVU’s Needle QA task directly evaluates video needle-in-a-haystack question answering. In this subcategory, DualStream substantially outperforms SimpleStream and performs similarly to the Visual-only baseline. This may be because in a query-agnostic streaming setting, the system does not know in advance which detail will be queried, and the particular needle is probably not aligned with the evolving episodic context.
W2. Comparison with retrieval-based memory approaches
DualStream studies how evolving episodic understanding can improve subsequent visual retention. Retrieval is complementary to this mechanism. Episodic feedback guides which incoming evidence enters memory, while retrieval determines which stored historical information is accessed.
Our current design maintains a compact state from which questions can be answered directly. Video-RAG’s original pipeline includes question-conditioned processing of offline video, giving it different access conditions. StreamRAG provides a direct streaming comparison, but we could not find a public implementation.
During rebuttal, we reimplemented StreamRAG based on its paper and supplementary material, including its three core modules (SES, KEA, and DKI). We followed the reported hyperparameters and prompts, and used the same backbone, streaming settings and evaluation interface as DualStream.
DualStream achieves 17.44 percentage points higher accuracy than our StreamRAG implementation. This adds a direct QA comparison with a retrieval-based streaming memory architecture. DualStream also avoids a historical memory bank that grows with video duration and an additional retrieval stage at answer time.