acceptodds
Under review as a conference paper at ICLR 2027

DualStream: Episodic Feedback Shapes Visual Memory for Long-Horizon Streaming Video Understanding

Abstract

Vision-language models commonly construct video context offline from a complete video. Streaming video understanding instead requires context to be built under resource constraints as frames arrive, often before the user question is known. Many streaming methods update this context based on visual redundancy or local changes, but visual novelty alone does not determine what is worth retaining. The importance of incoming details also depends on earlier events and can change as the episode unfolds. To address this gap, we present DualStream, a training-free framework that uses evolving textual episodic memory to guide visual retention. The visual stream maintains a compact token pool, which the textual stream periodically reads to update a running summary of the observed video. Each update also produces a future focus statement that guides the ranking of visual tokens from subsequent frames. Newly retained evidence then informs the next episodic update, forming a closed loop that adapts visual selection as the video develops. On the long subset of Video-MME, DualStream reaches 60.0% accuracy, exceeding the strongest online baseline by 4.7 percentage points while dropping 86.1% of visual tokens. Beyond these gains, at the same visual token budget, DualStream outperforms both visual memory alone and combined visual and textual memory without feedback.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, vision-language models, long video understanding, multimodal memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QpyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DualStream, a training-free framework for long-horizon streaming video understanding that integrates compact visual memory with textual episodic memory through a closed-loop feedback mechanism. The key idea is to use an evolving episodic summary to generate a future focus that guides subsequent visual token selection, rather than relying solely on visual redundancy. The framework is question-agnostic and operates with frozen VLM backbones.

The authors evaluate DualStream on multiple long-video and streaming QA benchmarks. On Video-MME Long, it achieves 60.0% accuracy while discarding 86.1% of visual tokens, outperforming the strongest reported online baseline by 4.7 percentage points. Ablation studies demonstrate that episodic feedback improves accuracy over visual-only memory by 2.22 percentage points under the same token budget. Experiments with Qwen2.5-VL-7B and InternVL3-8B also demonstrate transferability without additional training.

The primary contribution is improving long-horizon video understanding under constrained visual memory budgets. However, the evaluation mainly establishes improvements in aggregate QA accuracy, rather than directly characterizing long-term information retention, task-specific capabilities, or practical advantages over retrieval-based memory architectures.

Critical strengths

Training-free framework.
The proposed framework is training-free and does not require updating the parameters of the underlying VLM. This is a practical advantage because it avoids additional training costs and allows the proposed episodic feedback mechanism to be transferred to different VLM backbones. The experiments with Qwen2.5-VL-7B and InternVL3-8B provide initial evidence of this transferability, although model-specific adaptation and calibration are still required.

Critical weaknesses

1. Limited experimental analysis of long-horizon understanding capabilities.
The evaluation primarily reports overall accuracy on several video understanding benchmarks. However, these aggregate results do not sufficiently clarify which specific capabilities benefit from the proposed episodic feedback mechanism and where its limitations lie. A more fine-grained evaluation across task subcategories, such as the analyses presented in QViC-MF (CVPR 2026), would be valuable. In particular, Needle-in-a-Haystack (NIAH) evaluations are important for assessing whether the proposed selective visual memory can preserve rare but potentially critical information over long time horizons. Without such analyses, the extent to which DualStream improves long-term information retention remains unclear.

2. Insufficient comparison with retrieval-based memory approaches.
One of the main practical motivations of DualStream appears to be accurate long-horizon streaming video understanding under constrained memory resources. However, the paper does not directly compare the proposed framework with relevant retrieval-based approaches, such as Video-RAG (NeurIPS 2025) and StreamRAG (CVPR 2026), or temporal graph-based memory approaches. Although these methods may differ in their assumptions and memory architectures, they address closely related challenges in efficiently retaining and accessing information from long video streams. Comparisons under clearly defined and comparable memory, latency, and accuracy constraints would help establish the practical advantages of DualStream. Without such comparisons, its engineering value relative to alternative memory architectures is not sufficiently demonstrated.

Questions for authors

1. Potential for model-agnostic generalization

One of the strengths of DualStream is its training-free design. However, Appendix D indicates that applying the framework to different VLM backbones requires architecture-specific modifications and hyperparameter calibration. Could the authors clarify whether the proposed episodic feedback mechanism could be extended to a model-agnostic framework without relying on backbone-specific attention extraction or token representations? Achieving such generality would substantially strengthen the novelty and practical applicability of the proposed approach.

2. Real-time streaming feasibility

The reported mean visual-state update latency is 625.95 ms on an NVIDIA A100, corresponding to approximately 1.6 fps for this component alone. While this supports the 1-fps setting, can DualStream sustain real-time processing when visual and episodic memory updates run concurrently? How does the effective throughput change under higher frame sampling rates or substantially longer streams?

3. End-to-end computational efficiency

Although DualStream discards 86.1% of visual tokens, the additional attention scoring and episodic memory updates introduce computational overhead. How does the total GPU memory consumption and end-to-end computational cost compare with the visual-only baseline under identical conditions? Does the improvement in QA accuracy justify the additional computational overhead?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Qpyx (1/2)

We thank the reviewer for recognizing our training-free design and contribution. We address the two weaknesses and the three questions below.

W1. Capability analysis and long-term information retention

To examine performance across capabilities, we added task-level MLVU results for the Visual-only baseline during rebuttal and compared them with the DualStream and SimpleStream results from the original submission. Overall accuracy is computed over all evaluation questions.

MethodPlot QANeedle QAEgoCountOrderAnomaly recognitionTopic reasoningOverall
DualStream (%)73.6578.3162.5039.8166.4175.0087.8370.38
SimpleStream (%)45.6450.9947.1616.9947.8863.5072.6249.22
Visual-only baseline (%)73.2878.0362.5031.0763.7170.5087.8368.68

The largest gains over Visual-only occur in counting, anomaly recognition, and action ordering. A possible explanation is that episodic memory explicitly records important events and their relationships for later question answering, while the generated focus guides subsequent visual selection toward semantically relevant evidence. In contrast, the visual-only baseline tends to filter out visually repetitive but semantically meaningful content.

MLVU’s Needle QA task directly evaluates video needle-in-a-haystack question answering. In this subcategory, DualStream substantially outperforms SimpleStream and performs similarly to the Visual-only baseline. This may be because in a query-agnostic streaming setting, the system does not know in advance which detail will be queried, and the particular needle is probably not aligned with the evolving episodic context.

W2. Comparison with retrieval-based memory approaches

DualStream studies how evolving episodic understanding can improve subsequent visual retention. Retrieval is complementary to this mechanism. Episodic feedback guides which incoming evidence enters memory, while retrieval determines which stored historical information is accessed.

Our current design maintains a compact state from which questions can be answered directly. Video-RAG’s original pipeline includes question-conditioned processing of offline video, giving it different access conditions. StreamRAG provides a direct streaming comparison, but we could not find a public implementation.

During rebuttal, we reimplemented StreamRAG based on its paper and supplementary material, including its three core modules (SES, KEA, and DKI). We followed the reported hyperparameters and prompts, and used the same backbone, streaming settings and evaluation interface as DualStream.

MethodCorrect / TotalVideo-MME Long accuracy (%)Source
StreamRAG383 / 90042.56our reimplementation during rebuttal
DualStream540 / 90060.00main result in the original submission

DualStream achieves 17.44 percentage points higher accuracy than our StreamRAG implementation. This adds a direct QA comparison with a retrieval-based streaming memory architecture. DualStream also avoids a historical memory bank that grows with video duration and an additional retrieval stage at answer time.

AuthorsOpenReview ↗

Response to Reviewer Qpyx (2/2)

Q1. Potential for model-agnostic generalization

The episodic feedback loop transfers across backbones: retained evidence updates episodic understanding, and the resulting focus guides subsequent visual retention. Our current implementation translates that focus into token scores through the backbone’s internal attention.

Appendix D in the original manuscript describes the adapters needed to map this shared selection rule onto each backbone’s visual-token layout, position encoding, and attention interface. These adaptations keep all model parameters frozen. Layer selection and compression calibration use unlabeled visual inputs and generic focus prompts.

These adaptations accommodate differences between backbones while preserving the same episodic feedback mechanism.

Q2. Real-time streaming feasibility

Our target setting is a 1-fps input stream, which is a typical setting used in streaming video understanding research. As reported in Appendix C.3 of the original submission, the mean duration of the complete visual-state update is 625.95 ms, and its p99 latency is 970 ms, both within the one-second input interval.

Episodic generation operates at a much lower frequency and accounts for 2.46% of video duration. Visual selection uses the most recently completed focus while a new episodic update is being generated. Thus, the visual stream has no dependency on the completion of the new focus. Together, these measurements support the average processing budget at 1 fps.

Sampling rates above our default of 1 fps require a larger computational budget and make processing backlogs more likely. Longer videos do not affect the throughput of visual updates, since these updates process only newly arrived visual tokens and the computation is independent of the elapsed video duration.

Q3. End-to-end computational efficiency

The main additional per-frame operation over Visual-only is attention scoring and selection, which takes 177.22 ms on average in the original submission’s runtime breakdown. Sparse episodic memory generation introduces a further cost, with cumulative generation time accounting for 2.46% of processed video duration. Under real-time input, elapsed processing time is largely determined by video duration when processing keeps pace with incoming frames. Per-frame updates use the available time between frame arrivals. Finishing a frame earlier within that interval does not increase ingestion throughput.

During rebuttal, we also measured GPU memory using the same profiling configuration as the original runtime analysis. Each measurement covers processing a complete video and answering its three questions, including model weights and resident memory. We report the mean and maximum across per-video peaks. All values are in GiB.

ConfigurationPeak allocated: meanPeak allocated: max
Visual-only22.5925.39
DualStream22.8625.66

DualStream increases both the mean and maximum allocated peaks by 0.27 GiB. The additional per-frame computation mainly involves attention scoring on newly arrived visual tokens. Both methods retain model weights and visual memory, which account for the bulk of allocated GPU memory.

These measurements indicate a modest increase in allocated GPU memory and additional computation from attention-based selection and episodic generation, alongside accuracy gains of 2.22 percentage points on Video-MME Long and 1.70 points on MLVU.


We thank the reviewer for these constructive suggestions. We will include the new experimental results and corresponding clarifications in the revision.

Thank you for the additional experiments and explanations addressing my questions. However, I remain concerned that the effectiveness of episodic feedback for long-horizon video understanding, a central motivation of DualStream, has not been sufficiently demonstrated.

In the task-level MLVU results, DualStream shows a relatively modest overall improvement over the Visual-only baseline (+1.70 percentage points), while the improvement on Needle QA is almost negligible (+0.28 percentage points). Although the gains in some other task categories are encouraging, these results do not clearly establish that episodic feedback improves long-term information retention, particularly for rare but potentially important information.

Thank you for your response.

As described in the original MLVU paper, Needle QA is constructed by randomly inserting a short clip from another video into a background video. Manual verification further ensures that the needle scene does not appear in the background video. This construction gives the inserted needle no established connection to the surrounding episode. In a query-agnostic streaming setting, guidance derived from the surrounding episode is therefore not expected to prioritize the needle.

The gains in counting, anomaly recognition, and action ordering demonstrate the benefits of episodic feedback for understanding events across a long video, where visually similar observations can have different meanings in the unfolding episode.

Thank you again for your consideration. We would appreciate your thoughts on this distinction between retaining independently inserted details and understanding events within an unfolding episode.

Reviewer_otLnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents DualStream, a training-free, streaming video understanding framework which constructs memory before the user’s question is known. The framework maintains both a compact visual token pool and textual episodic memory which contains a continuously updated summary and a follow-up instruction. Newly arrived visual tokens are first filtered via Differential Token Drop, then ranked based on text-to-visual attention from the latest available focus. At sparse observation anchors, the frozen backbone update both textual components using the retained visual memories and previous summary, threrby forming a feedback loop that guides subsequent visual retention. Evaluation covers end-of-stream and timestamped video question answering, together with feedback mechanism ablation and efficiency measurements, which to some extent demonstrated the effectiveness of the approach.

Critical strengths

The experiments described in the paper demonstrate the effectiveness of the method, to a certain extent. The paper reports a statistically supported accuracy improvement under a matched visual token budget. Moreover, matched retention budget and video-clustered analysis strengthen the evidence that the implemented guidance signal is practically useful within the evaluated process.

Critical weaknesses

The selection criterion in this paper lacks sufficient validation. The layer-selection analysis in Appendix A.3 generally follows the existing experimental paradigm: comparing cross-modal attention across decoder layers across different decoder layers and selecting for visual token selection. FlexSelect (Yunzhu Zhang et al., FlexSelect: Flexible Token Selection for Efficient Long Video Understanding. NeurIPS 2025) already studies this question using recovery of known relevant tokens and downstream QA comparisons across layers. As a key improvement, DualStream employs a different label-free criterion, selecting a certain layer by minimizing the mean pairwise correlation of attention-score vectors across focus statements on fixed visual inputs. Lower cross-focus correlation indicates text sensitivity, but this alone does not prove that the resulting attention changes identify semantically appropriate visual evidence. Table A.1 reports correlations rather than selection quality, and the paper does not report whether layers with lower correlation achieve better evidence retention or QA accuracy under matched visual token budgets. Consequently, compared with existing layer-by-layer analyses, the layer-selection component seems to offer limited additional insight beyond prior layer analyses, and its discriminatory criteria have yet to be fully validated.

The most crucial missing comparison is the use of the dynamically updated video summary as queries for visual token selection. Table 2 compares the full method (60.0%) with a no-interaction variant (57.1%) that uses the fixed generic query “What is happening in this clip?” throughout the video. Although this improvement is supported by statistics, it does not show whether future-focus queries are more effective than dynamic summary queries. The method maintains a continuously updated running summary of the observed video, but does not report a variant that uses a condensed version of that summary to guide visual token selection. This controlled comparison should be carried out by updating the summary queries at the same observation anchors, using a comparable query length. It seems that no experiment has been conducted to compare future focus with the updated summary. Without this comparison, the paper does not establish what its separate future-focus state adds beyond using an updated summary to guide selection, leaving a central design choice insufficiently validated. The added value of the separate future-focus state therefore remains insufficiently established

Questions for authors

Is there any evidence to suggest that the layers with lower cross-focus attention correlation achieve better evidence retention or QA accuracy given the same visual token budget?

How does using ‘next_focus’ compare with a dynamically updated summary as the selection query?

4 comments
AuthorsOpenReview ↗

Response to Reviewer otLn (1/2)

We thank the reviewer for recognizing our accuracy gains under matched budgets and the practical usefulness of our guidance signal. DualStream’s central contribution is episodic feedback: retained visual evidence updates the model’s understanding of the episode, which then guides subsequent visual retention. Within this feedback loop, the choice of attention layer and the specific formulation of the guidance query are flexible and not restricted to a particular implementation. We address the two questions below.

Attention-layer selection and downstream QA

Our choice of a middle decoder layer draws on existing studies. FlexSelect, highlighted by the reviewer, also reports that an intermediate layer provides stronger retrieval of relevant visual tokens and better downstream QA than shallow or final layers in its experiments. We use a simple text-sensitivity heuristic to calibrate the extraction layer for our backbone. With visual inputs fixed, we measure the mean pairwise correlation between attention-score vectors induced by different focus statements. Lower correlation indicates greater responsiveness to the textual guidance. This probe selects the zero-indexed L14 of Qwen2.5-VL-7B’s 28 decoder layers, consistent with prior findings favoring intermediate layers. This heuristic serves as a practical configuration choice within DualStream rather than a separate methodological contribution. The core feedback mechanism does not depend on this particular layer-selection procedure.

To examine whether this diagnostic is consistent with downstream performance, we added layer ablations on Video-MME Long during rebuttal. We varied the attention extraction layer while keeping the backbone, visual-token retention budget, and other settings unchanged. The p-values are from individual two-sided exact McNemar tests. Answer transitions are from the L14 baseline to each layer variant.

LayerAccuracy (%)Correct to wrong vs. L14Wrong to correct vs. L14p-valueSourceCross-focus correlation
L057.5656340.02630added during rebuttal0.9845
L1158.781980.05224added during rebuttal0.8108
L1460.00———main results in the original submission0.5937
L1959.8916151.00000added during rebuttal0.6574
L2757.3361370.01969added during rebuttal0.9450

The two layers with the lowest cross-focus correlations, L14 and L19, achieve the highest accuracies among the tested layers, differing by only one correct answer. The shallow and final layers have higher correlations and lower accuracies, with L11 falling between these groups. This pattern provides downstream support for the text-sensitivity heuristic and for L14 as a reasonable configuration. The similar performance of L14 and L19 also shows that effective feedback can be implemented with more than one middle-layer configuration.

AuthorsOpenReview ↗

Response to Reviewer otLn (2/2)

Separate focus versus a dynamically condensed summary

In the original DualStream design, the running summary preserves broad cumulative understanding of the observed video, while the focus expresses selected aspects of this understanding as a compact query for subsequent visual selection. During rebuttal, we extended the focus statement audit in Section 4.3 to examine the relationship between the self-generated summary and focus. We used DeepSeek-Flash-1017 as an independent judge on the same 3,846 focus statements from Video-MME Long. The judge received only each focus and its contemporaneous cumulative summary. Because a focus statement can contain multiple observation targets, we asked the judge to separate each statement into distinct content units and classify their relationship to the summary. This yielded 8,698 content units.

Relationship of focus content to the summaryContent unitsPercentage (%)
Already contained in the summary3,43939.54
Reasonable observation extensions based on summary cues5,18059.55
No reasonable connection to the summary530.61
Uncertain260.30

In 925 of the 3,846 focus statements (24.05%), all content units were judged to be contained in the corresponding summary. More commonly, a focus combines selected summary content with reasonable observation extensions. For example, a summary mentions many scenes and objects including a toy next to a cat, but the corresponding guidance is to watch for interaction between the toy and the cat. This illustrates selective condensation with a clear observational emphasis. These results support interpreting focus as a selective condensation of current episodic understanding together with observation targets motivated by existing cues.

Compressing the running summary is an alternative way to construct guidance within the episodic feedback mechanism. We added the requested condensed-summary control on Video-MME Long, using the same streaming settings and visual-token retention budget. At each anchor, the model updates the full summary and generates the original focus using the existing prompt. The same frozen backbone then compresses the summary through a separate text-only call that receives only the summary and a target length L, defined by the token length of the concurrently generated focus. If the compressed output exceeds 1.2L tokens, we provide token-count feedback and retry up to three times. Outputs that remain overlength are truncated to this cap. The condensed summary replaces the focus as the query for subsequent visual selection. The full summary remains available for later episodic updates. The experiment runs online, with each summary update based on evidence retained under that variant’s preceding guidance queries.

MethodCorrect / TotalAccuracy (%)Correct only with this methodMcNemar p-valueSource
DualStream540 / 90060.0021—main results in the original submission
DualStream w/ condensed summary as query529 / 90058.78100.0708added during rebuttal

The results are shown above. Overall, 3.60% of the compressed summaries required truncation. The original focus achieves 1.22 percentage points higher accuracy. The comparison supports retaining the original focus as a reasonable implementation, while the condensed-summary variant provides an alternative way to express episodic guidance within the same core mechanism.


We thank the reviewer for these constructive suggestions. We will include the new experiments, audit details, and corresponding discussion in the revision.

Thank you for the additional experiments, which address my questions about layer selection and the comparison with a dynamic summary. However, I still have reservations about the overall methodological contribution and maintain my rating.

Thank you for confirming that the additional experiments address your questions.

Our central contribution is the feedback loop between evolving episodic understanding and visual retention: retained evidence updates the model’s understanding of the episode, which then guides what to retain from subsequent frames.

We would appreciate knowing whether your remaining concern is that DualStream is too close to existing methods, that the feedback mechanism needs stronger experimental validation, or something else. Thank you again for your response.

Reviewer_kHVROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DualStream, a training-free framework for question-agnostic streaming video understanding. It maintains a compact visual token pool and a periodically updated textual episodic memory. Each update produces a running summary and a future-focus statement, which guides attention-based selection of visual tokens from subsequent frames after temporal redundancy filtering. Experiments cover long-video QA, timestamped streaming QA, and transfer to a second backbone. The central finding is that episodic feedback improves accuracy over visual-only memory and combined visual/textual memory without feedback at matched visual-token budgets.

Critical strengths
  1. The problem and design are well motivated. When future questions are unknown, deciding which observations to preserve is important. Using an evolving semantic account to guide subsequent visual retention is a sensible extension of redundancy filtering. The framework operates with frozen backbone weights.
  2. The central result is supported by useful controls. On Video-MME long, DualStream reaches 60.0%, compared with 57.8% for visual-only memory and 57.1% for visual/textual memory without feedback, at the same reported visual-token drop rate. Appendix B.1 provides paired comparisons and confidence intervals clustered by video. These results support a modest but meaningful benefit from feedback. The improvement on InternVL3-8B provides additional evidence of transferability.
Critical weaknesses
  1. The incremental contribution needs stronger positioning and targeted comparisons. Two relevant approaches merit closer consideration. rLiVS uses recurrent visual memory and caption attention for token selection, while SURGE uses temporal prediction error as a training-free selection signal. Neither is identical to DualStream, but they raise an important question: how much benefit comes specifically from evolving episodic guidance, beyond existing semantic or novelty-based selection? The fixed generic-query control is useful, but does not fully separate semantic specificity from dependence on episodic history. A targeted comparison under the same backbone and retention budget would clarify the contribution.
  2. The coverage and recovery properties of a single focus statement remain unclear. One sentence can contain multiple targets, but it still compresses selection priorities into one short instruction despite unknown future questions and potentially concurrent events. This creates a trade-off between focused retention and diverse evidence coverage. Appendix B.4 illustrates a focus/question mismatch without quantifying its impact or subsequent recovery. Recovery is particularly relevant because DTD excludes candidates before focus-guided ranking: an initially omitted static detail may remain unavailable to a later focus. Subsequent episodic updates read already filtered evidence and could reinforce earlier omissions. A controlled analysis of focus mismatch or topic shifts, including multiple complementary foci under the same total retention budget, would strengthen the design justification.

References

[1] Dorovatas, E., Seifi, S., Gupta, G., and Aljundi, R. Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs. NeurIPS, 2025.

[2] Tang, C., Ek, S., Koch, D., Mullins, R., Weddell, A., and Chauhan, J. SURGE: Surprise-Guided Token Reduction for Efficient Video Understanding with VLMs. ICLR, 2026.

Questions for authors

How does episodic feedback improve on related selection strategies? Please discuss rLiVS and SURGE and clarify the conditions under which comparisons are meaningful. A matched comparison with caption-based selection or a causal surprise-based selector would be informative.

Why represent guidance with one short focus statement? Have multiple complementary foci been tested under the same total visual-token budget? How does the system recover when important evidence falls outside the current focus, particularly when that evidence is subsequently filtered as temporally redundant?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kHVR (1/2)

We thank the reviewer for recognizing our motivation, design, and central result. We address the two questions below.

Added value of episodic guidance over existing selection strategies

DualStream’s contribution is to explicitly maintain an evolving episodic state and feed it back into subsequent visual retention. rLiVS and SURGE provide relevant comparisons that help clarify this contribution.

rLiVS conditions current-clip caption generation on recent visual memory, then selects tokens using attention collected during that generation. DualStream instead converts accumulated episodic understanding into an explicit focus query that is reused for subsequent visual updates. This separates frequent visual retention from sparse text generation, allowing incoming frames to be selected without requiring a new caption to be generated for each visual update. The retained visual evidence also remains available for answering questions, whereas rLiVS’s main answering pipeline uses accumulated captions.

SURGE’s base selector measures novelty through prediction error in recent visual features. DualStream additionally maintains a semantic account of the episode, forming a feedback loop in which retained evidence updates episodic understanding and that understanding guides later retention.

During rebuttal, we added end-to-end Video-MME evaluations using the official rLiVS and SURGE repositories, with the same backbone and visual-token retention budget, and streaming settings aligned with those of DualStream. For SURGE, whose published Video-MME results use an offline protocol, we implemented a streaming adaptation.

MethodSelection SignalVideo-MME Overall (%)Video-MME Long (%)SourceIntermediate Generation Time / Video Duration (%)
rLiVScaption attention51.4846.89our matched reimplementation during rebuttal, using the official rLiVS repository26.45
SURGE (streaming adaptation)visual surprise63.3756.33our matched streaming adaptation during rebuttal, using the official SURGE repositoryno generation
DualStream (ours)self-generated focus67.3060.00main results in the original submission2.46

These results complement our existing ablations, and show that DualStream’s gains extend to comparisons with caption-based and surprise-based selection approaches. We also measured intermediate text-generation cost as cumulative generation time divided by processed video duration. DualStream’s lower generation overhead supports the computational value of separating visual retention and semantic updates across two timescales.

AuthorsOpenReview ↗

Response to Reviewer kHVR (2/2)

Focus coverage and evidence recovery

We use a single short focus to distill current episodic understanding into a compact selection query that is reused across frequent visual updates, controlling repeated scoring overhead. Compact guidance can still express multiple directions. During rebuttal, we extended the focus audit in Section 4.3 using DeepSeek-Flash-1017 to identify distinct selection priorities in the 3,846 focus statements.

Distinct priorities per focus statement1234>=5
Number of focus statements1,5011,921334864
Percentage (%)39.0349.958.682.240.10

The audit identifies at least two distinct priorities in 60.97% of the statements, showing that a compact focus can already include more than one direction.

Separate focus statements offer an alternative way to express complementary priorities and allocate retention capacity. During rebuttal, we evaluated this extension on Video-MME Long under the same total visual-token budget. We modified the prompt to generate at least one focus statement, allowing the model to determine their number. Each statement independently scored the visual candidates. Tokens were selected in round-robin order, skipping duplicates until the common retention budget was filled.

MethodCorrect / TotalAccuracy (%)Correct only by this methodSource
DualStream540 / 90060.0020main results in the original submission
DualStream w/ multiple foci528 / 90058.678added during rebuttal

The number of focus statements generated per update was distributed as follows.

Focus statements generated12345>=6
Number of updates1,0191,213766385244219
Percentage (%)26.5031.5419.9210.016.345.69

The multiple-focus variant does not improve downstream QA in this setting. This result supports retaining the original single-focus design while recognizing separate focus statements as a possible extension.

Regarding recovery, evidence omission is inherent to streaming video understanding with bounded memory and unknown future questions. The relevant criterion is how effectively useful information is preserved under a fixed budget. Our existing ablations show that episodic feedback improves QA accuracy over both visual-only memory and combined visual–textual memory without feedback, supporting its effectiveness in addressing this trade-off.

Subsequent focus updates can redirect the selection of incoming visual evidence. Previously omitted details can be retained if they reappear among the DTD candidates. If a detail is absent from retained visual or textual memory and remains excluded by DTD, focus updates alone cannot recover it. Recovery in this case would require an external archive, such as stored frames or richer historical representations, together with additional retrieval mechanisms. Such archive-based recovery is a complementary extension beyond DualStream’s core mechanism of using evolving episodic understanding to guide visual retention under a compact memory budget.


We thank the reviewer for these constructive suggestions. We will include the details of the new experiments and the corresponding discussion in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.