acceptodds
Under review as a conference paper at ICLR 2027

StreamBank: Unified Generation–Memory Inference for Efficient Streaming Video Understanding

Abstract

Streaming video understanding requires Video Large Language Models (Video-LLMs) to generate timely responses while preserving relevant historical information from continuous video streams. However, autoregressive response generation incurs substantial decoding latency, while continuously growing visual–textual histories introduce increasing computation and memory costs. Even with speculative decoding, candidate generation often remains sequential, while existing memory policies typically rely on visual similarity or recency rather than prediction-relevant evidence to retain historical events. To address these limitations, we propose StreamBank, a unified inference framework that uses target-model verification to accelerate response generation and guide historical event retention. StreamBank comprises two complementary components. First,Reuse-Augmented MTP Speculative Decoding predicts multiple future tokens in parallel and reuses previously verified responses as candidates when temporal continuity permits, reducing drafting overhead while retaining target-model verification. Second, Speculation-Guided Event Memory repurposes verifier-gain and correction signals from the same verification process to guide event promotion, retention, and eviction within a fixed-capacity memory bank. Together, these components connect efficient generation with prediction-aware memory management through shared verification feedback. Experiments on ProactiveVideoQA, OVO-Bench, and StreamingBench show that StreamBank largely preserves streaming video understanding performance under substantial context compression while achieving up to a decoding-throughput speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Video Large Language Models, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hVoJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamBank combines speculative decoding and event memory for streaming video understanding. It uses parallel multi-token prediction and reuses previous responses to propose draft tokens. A frozen target model verifies these tokens. Feedback from verification guides the retention and eviction of historical events in a fixed-capacity memory. Experiments report a 2.01× decoding-throughput speedup and improved average scores on three streaming video benchmarks.

Critical strengths
  1. StreamBank uses verification feedback for both generation acceleration and event memory management. Table 2 reports a 2.01× decoding-throughput speedup with improved overall pAUC. Table 3 shows better event-recall accuracy than FIFO and visual similarity under the same memory budget. These results support the practical value of the combined framework.
Critical weaknesses
  1. The value of verification feedback for memory selection is not sufficiently established. Table 3 compares retention policies but does not isolate the gain and correction signals. It remains unclear whether these signals select useful events better than simpler task-relevance scores. This weakens the evidence for the paper’s main contribution.
  2. The efficiency evaluation is incomplete. The reported 2.01× speedup measures decoding throughput. The paper does not report end-to-end latency that includes video processing, memory updates, and event-caption generation. The actual benefit for real-time interaction therefore remains unclear.
Questions for authors
  1. Do you have ablations showing the separate contributions of the gain and correction signals? How do they compare with a simpler task-relevance score under the same memory budget?
  2. What is the end-to-end speedup when video processing, memory updates, and event-caption generation are included?
5 comments
AuthorsOpenReview ↗

Answer to Reviewer hVoJ

[W1 & Q1] Insufficient validation of verification feedback for memory selection.

Thank you for this suggestion. We agree that the contributions of the signals obtained from target-model verification should be assessed separately. We therefore ablate Verifier Gain and Correction Evidence on ProactiveVideoQA while keeping memory capacity and decoding settings unchanged. The baseline without either verification-derived signal still retains Novelty and Direct-response Support.

FeedbackOverall pAUC
Neither29.5
Verifier Gain only30.7
Correction Evidence only30.8
Full feedback31.0

Removing both signals reduces overall pAUC from 31.0 to 29.5; adding Verifier Gain or Correction Evidence alone increases it to 30.7 and 30.8, respectively, with the best result obtained by combining both. This shows that verification-derived feedback provides additional improvements beyond novelty and direct-response support, offering direct evidence for the effectiveness of generation–memory coupling.

[W2 & Q2] Incomplete end-to-end efficiency evaluation.

Thank you for raising this question. To assess overall system efficiency, we conduct additional end-to-end experiments comparing AR, MTP, and MTP + Reuse under the same evaluation workload.

MethodE2E SpeedupTTFT (ms)Latency (ms)
MMDuet21.00×225.7 / 1203.81033.8 / 3757.5
MTP2.71×108.7 / 320.4899.7 / 3361.3
MTP + Reuse2.85×108.1 / 317.2896.6 / 2986.3

E2E Speedup is computed as the ratio of total processing time under the same workload, including video processing, memory updates, event-caption generation, and response generation. TTFT and response latency measure per-response performance and are reported as P50/P95.

MTP achieves a 2.71× end-to-end speedup over MMDuet2, which further increases to 2.85× with response reuse. These results demonstrate that our proposed mechanisms improve overall inference efficiency, with benefits extending beyond token-level decoding acceleration.


Thank you again for your insightful comments. We hope our additional results and implementation details have addressed your concerns, and we would greatly appreciate your reconsideration of our work.

Thanks for running these extra experiments. The ablation clears up my main concern about whether verifier feedback helps memory selection. I have one follow-up on the timing results. Could you share which stages account for most of the 2.85× speedup across the full workload? That would help me connect the overall speedup to the per-response latency numbers.

AuthorsOpenReview ↗

Answer to Reviewer hVoJ

Thank you for your follow-up. The overall speedup primarily comes from the response generation stage. On the one hand, StreamBank compresses historical context through event memory management, reducing the number of context tokens that need to be processed and thereby lowering prefill overhead. On the other hand, MTP and response reuse further accelerate autoregressive generation. Consequently, the response generation time decreases from 28.8 s/clip with AR to 10.3 s/clip with MTP, and further to 9.5 s/clip with MTP + Reuse, making it the primary contributor to the end-to-end speedup.

Reviewer_NyYtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes StreamBank, a unified framework for improving decoding efficiency and memory management in streaming video understanding. It combines reuse-augmented multi-token speculative decoding with a speculation-guided event memory mechanism that uses draft-verifier discrepancies to prioritize historical information. The framework also allows response reuse across visually similar observations and maintains a bounded event memory for long-term context. Experiments on OVO-Bench, StreamingBench, ProactiveVideoQA, and long-video benchmarks report improvements in accuracy and decoding throughput over the MMDuet2 backbone.

Critical strengths

The paper is clearly written and is easy to follow. The work is well-motivated and presents a unified framework for improving decoding efficiency and memory management in streaming video understanding. The evaluation spans multiple streaming and long-video benchmarks.

Critical weaknesses
  • W1. [Experiments] I have several concerns regarding the experimental evaluation.

    • W1.1. [Missing recency baseline] The paper highlights the limitations of recency-based memory, yet such baselines are absent from the main tables. Recent works [a, b] highlight important limitations in evaluating streaming video understanding on OVO-Bench and StreamingBench, where a straightforward recent-frame baseline can be surprisingly strong. In particular, SimpleStream [a] achieves 73.47 on OVO-Bench Real-Time with Qwen2.5-VL-3B and only the last 4 frames, compared to StreamBank's 62.7. A controlled comparison using the same backbone is necessary to establish whether the additional complexity of the proposed method is justified.
    • W1.2. [Missing newer baselines] Tab. 1 omits several relevant methods, including Streamo [c], VST [d], ThinkStream [e] and HERMES [f] which have 3B variants and report results on OVO-Bench and StreamingBench. Several of these methods report stronger performance than StreamBank, making their omission particularly concerning.
    • W1.3. [MMDuet2 reproduction] Since the primary model for comparison is MMDuet2, the original paper reports substantially higher ProactiveVideoQA scores (39.8) than the reproduced baseline in Tab. 2 (30.1). I understand that the original evaluation uses GPT-4.1 as the judge, whereas this paper uses Qwen3-32B, which may explain some of the discrepancy. However, the large difference makes it difficult to contextualize the reported improvements. Reporting results under the original evaluation protocol, or running the open-source judge across multiple seeded runs, would make the takeaways convincing.
    • W1.4. [Insufficient component ablations] The ablations do not sufficiently isolate the contributions of the proposed components. Tab. 2 evaluates response reuse on ProactiveVideoQA, but lacks comparisons with established speculative decoding strategies. Tab. 3 only ablates memory-retention policies on OVO-Bench EPM, showing an incremental gain over FIFO without isolating speculation-guided feedback from the other retention heuristics. Given that the main results in Tab. 1 focus on real-time subtasks, where recency baselines will dominate, stronger ablations on historical-memory tasks are particularly important.
  • W2. [Novelty] The methodological novelty appears somewhat limited. Sec. 3.3 largely combines established speculative decoding techniques, while Sec. 3.4 relies on heuristic alignment scores and weighted event-retention rules. While incremental methodological contributions are perfectly reasonable, they should be supported by convincing empirical improvements, which are not sufficiently demonstrated in the current evaluation (see W1).

    • W2.1. [Necessity of training] The method introduces a two-stage training procedure for the adapter and MTP heads, while the response-reuse and event-memory policies themselves do not require training. It is unclear whether this additional training is necessary. A training-free variant on Qwen2.5-VL-3B-Instruct or MMDuet2 or comparison with training-free speculative decoding methods, such as SpecVLM [h], would help establish the benefit of the dedicated training.
  • W3. [Efficiency] The reported 2.01× speedup measures decoding throughput rather than end-to-end streaming latency, leaving its practical implications unclear. In a streaming setting (e.g., 1 fps input), the relevant metrics are time-to-first-token (TTFT) and response latency i.e. whether inference can keep pace with incoming frames without dropping them. Furthermore, the method freezes the visual context during generation, unlike full-duplex systems [i] that support concurrent perception and generation. As such, a real-world takeaway for streaming applications is missing that demonstrates the benefit of the 2.01× speedup.

[a] A Simple Baseline for Streaming Video Understanding, arXiv 2026.

[b] Don't Pause! Every Prediction Matters in a Streaming Video, NeurIPS 2026 E&D

[c] Streaming Video Instruction Tuning, CVPR 2026.

[d] Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously, ECCV 2026.

[e] Thinking in Streaming Video, arXiv 2026.

[f] HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding, ACL 2026.

[g] MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning, ICLR 2026.

[h] SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning, EMNLP 2025.

[i] MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction, arXiv 2026

Questions for authors

I have outlined my concerns and questions under Critical Weaknesses. I would appreciate it if the authors could address the above points in their rebuttal.

7 comments
AuthorsOpenReview ↗

Answer to Reviewer NyYt [part1/3]

We sincerely thank you for recognizing the clarity of the paper, the motivation and unified framework, and the breadth of evaluations across streaming and long-video benchmarks. We respond to each concern below.

[W1.1] Missing recency baseline.

Thank you for your suggestion. We agree that comparisons with recent-frame baselines are important for evaluating event memory management. We therefore provide additional OVO-Bench results covering both Real-Time Visual Perception and Backward Tracing, along with comparisons against more existing methods.

ModelReal-TimeBackwardAvg.
SimpleStream-3B73.542.958.2
MMDuet2-3B†62.043.652.8
StreamBank62.745.454.1

Although SimpleStream [a] achieves stronger real-time visual perception, StreamBank outperforms it by 2.5 points on Backward Tracing, which evaluates retrieval and reasoning over historical video events. Relative to our base model, MMDuet2-3B, StreamBank improves Backward Tracing from 43.6 to 45.4, including gains of 3.7 and 2.1 points on EPM and ASI, respectively.

Furthermore, with the same event-memory budget, StreamBank achieves 52.53% EPM accuracy, outperforming the recency-based FIFO baseline (51.18%). Although absolute performance is partly constrained by the underlying MMDuet2-3B model, these results suggest that event memory management improves historical information retention and long-range understanding beyond simple recency-based selection. We have added these comparisons to the appendix and clarified the distinction between real-time perception and historical event understanding.

[W1.2] Missing newer baselines.

Thank you for pointing out these missing methods. We agree that comparing against stronger recent streaming video understanding methods is important for a comprehensive evaluation. We have expanded our OVO-Bench comparison to include Streamo [b], VST [c], ThinkStream [d], HERMES [e], and other recent approaches, reporting both Real-Time Visual Perception and Backward Tracing.

ModelReal-TimeBackwardAvg.
Streamo-3B61.541.851.6
ThinkStream-3B67.052.359.7
VST-7B67.256.762.0
HERMES-4B72.355.063.7
MMDuet2-3B†62.043.652.8
StreamBank62.745.454.1

We acknowledge that StreamBank still trails stronger models on some metrics. This partly reflects its reliance on the MMDuet2-3B backbone, which constrains its absolute capability. Nevertheless, relative to the same base model, StreamBank improves Real-Time Visual Perception and Backward Tracing by 0.7 and 1.8 points, respectively, increasing the overall score from 52.8 to 54.1. These controlled improvements support the effectiveness of our event memory management. We have added the extended comparisons to the appendix.

[W1.3] MMDuet2 reproduction

Thank you for this suggestion. We agree that the substantial discrepancy between the original MMDuet2 scores and our reproduced baseline requires clarification and a robustness check.

We use MMDuet2 based on Qwen2.5-VL-3B as the base model and perform inference on a single NVIDIA A800 GPU. Whereas the original paper uses GPT-4.1 as its evaluator, our experiments adopt Qwen3-32B, which may partly explain the differences in absolute scores. Importantly, StreamBank is built and trained on MMDuet2 while keeping the target model frozen. We therefore have no incentive to underestimate MMDuet2, whose capabilities directly underpin our method. Our primary interest is the performance improvement over the base model under identical evaluation settings.

To further assess reliability, we perform additional evaluations with Qwen3-32B using three different judge seeds:

SeedMMDuet2StreamBank
4230.1531.00
12330.1231.04
66630.1231.07
Mean ± Std.30.13 ± 0.0231.04 ± 0.04

StreamBank consistently outperforms MMDuet2 across all three seeds, with an average gain of 0.91 points and low variance for both models. This indicates that the improvements remain stable across judge seeds, supporting the reliability of our controlled comparisons. We have included these results in the appendix.

AuthorsOpenReview ↗

Answer to Reviewer NyYt [part2/3]

[W1.4] Insufficient component ablations

Thank you for this constructive suggestion. We agree that more extensive component ablations and historical-memory evaluations are needed to isolate the contributions of decoding and memory management. We therefore add the following experiments.

(1) Comparison with existing speculative decoding methods. On the VAD subset of ProactiveVideoQA, we compare StreamBank with SpecVLM [f], Kangaroo [g], and Medusa [h] under the same frozen target model:

MethodDecode Speedup
Target AR1.00×
SpecVLM†1.22×
Kangaroo1.51×
Medusa2.05×
StreamBank (untrained)0.63×
StreamBank (w/o reuse)1.84×
StreamBank2.40×

† Our adaptation of SpecVLM using Qwen2.5-0.5B as the draft model.

StreamBank achieves a 2.40× speedup, outperforming SpecVLM (1.22×), Kangaroo (1.51×), and Medusa (2.05×). Its MTP-only variant reaches 1.84×, below Medusa; both employ multi-head token prediction, while Medusa uses tree-based verification to examine multiple candidate paths. By additionally exploiting temporal redundancy through cross-turn response reuse, StreamBank improves from 1.84× to 2.40×, highlighting the benefit of reuse beyond multi-head drafting.

(2) Ablation of verification-derived feedback. With memory capacity and decoding settings held fixed on ProactiveVideoQA, we separately evaluate Verifier Gain and Correction Evidence:

FeedbackOverall pAUC
Neither29.5
Verifier Gain only30.7
Correction Evidence only30.8
Full feedback31.0

Removing both signals decreases overall pAUC from 31.0 to 29.5, while adding either signal individually improves performance. Thus, verification feedback provides gains beyond the other heuristic signals used for memory selection.

(3) Historical-memory task evaluation. On OVO-Bench EPM, we add Recent only and StreamForest-style [i] as retention baselines:

Retention policyEPM accuracy (%)
Recent only50.2
FIFO51.2
Visual similarity49.8
Full history48.8
StreamForest-style51.5
StreamBank52.5

StreamBank achieves 52.53% EPM accuracy, outperforming Recent only (50.17%), FIFO (51.18%), and StreamForest-style (51.52%). Together, these experiments validate the effectiveness of our decoding mechanism, verification-derived feedback, and historical-event retention policy.

[W2] Limited methodological novelty.

Thank you for your feedback. While speculative decoding and memory management have been studied, StreamBank goes beyond simply combining them by leveraging target-model verification to connect generation acceleration with historical-event retention. Our contributions are twofold:

(1) Generation acceleration: Parallel MTP with cross-turn response reuse. Unlike existing methods focusing on single-turn decoding, StreamBank combines parallel token prediction with reuse of previously verified responses, exploiting both within-turn parallelism and cross-turn temporal redundancy. Although its MTP-only variant achieves 1.84× speedup, below Medusa's 2.05×, adding response reuse increases speedup to 2.40×, demonstrating benefits beyond conventional multi-head drafting.

(2) Memory management: Verification-guided event retention. Unlike recency- or similarity-based policies, StreamBank repurposes Verifier Gain and Correction Evidence from token verification to estimate event relevance and guide memory updates. This enables generation and memory management to share the same verification feedback rather than operate independently. Removing these signals reduces pAUC from 31.0 to 29.5. StreamBank also achieves 52.53% on OVO-Bench EPM, outperforming FIFO (51.18%) and StreamForest-style (51.52%).

Together, these results demonstrate the effectiveness of our unified generation–memory framework through shared target-model verification.

AuthorsOpenReview ↗

Answer to Reviewer NyYt [part3/3]

[W2.1] Necessity of training.

Thank you for your suggestion. We agree that the benefit of dedicated training should be examined. We first clarify that additional training is used only for parallel MTP drafting; response reuse and event memory management are training-free.

To address this concern, we compare an untrained StreamBank variant with existing speculative decoding methods, including the training-free SpecVLM, on the ProactiveVideoQA VAD subset under the same target model.

MethodDecode Speedup
Target AR1.00×
SpecVLM†1.22×
Kangaroo1.51×
StreamBank (untrained)0.63×
StreamBank (w/o reuse)1.84×
StreamBank2.40×

† Our adaptation of SpecVLM using Qwen2.5-0.5B as the draft model.

The untrained StreamBank achieves only 0.63×, whereas the trained MTP without response reuse achieves 1.84×, exceeding the training-free SpecVLM (1.22×) and Kangaroo (1.51×). Adding training-free response reuse further raises the speedup to 2.40×. These comparisons demonstrate the efficiency benefit of dedicated MTP training and its complementarity with training-free reuse.

[W3] Efficiency.

Thank you for highlighting this practical concern. We agree that decoding throughput alone does not capture the benefits for streaming inference. We have therefore added evaluations of end-to-end speedup, TTFT, and complete response latency.

MethodE2E SpeedupTTFT (ms)Latency (ms)
MMDuet21.00×225.7 / 1203.81033.8 / 3757.5
MTP2.71×108.7 / 320.4899.7 / 3361.3
MTP + Reuse2.85×108.1 / 317.2896.6 / 2986.3

E2E Speedup is computed as the ratio of total processing time under the same workload, including video processing, memory updates, event-caption generation, and response generation. TTFT and response latency measure per-response performance and are reported as P50/P95.

MTP + Reuse achieves a 2.85× end-to-end speedup over MMDuet2. P50/P95 TTFT decreases from 225.7/1203.8 ms to 108.1/317.2 ms, while complete response latency decreases from 1033.8/3757.5 ms to 896.6/2986.3 ms. Notably, median response latency falls below the one-second frame interval corresponding to 1 fps, indicating improved responsiveness. Although P95 latency remains above this interval, it decreases from 3757.5 ms to 2986.3 ms, showing an improvement in tail latency. We will continue exploring ways to reduce tail latency and improve real-time performance under continuous streaming input.

Regarding full-duplex interaction, we acknowledge that StreamBank freezes the visual context while generating a response and does not currently support concurrent perception and generation. Our work primarily focuses on inference efficiency in turn-based streaming video understanding, rather than full-duplex interaction. We plan to investigate asynchronous visual perception and response generation in future work.

[a] A Simple Baseline for Streaming Video Understanding, arXiv 2026.

[b] Streaming Video Instruction Tuning, CVPR 2026.

[c] Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously, ECCV 2026.

[d] Thinking in Streaming Video, arXiv 2026.

[e] HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding, ACL 2026.

[f] SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning, EMNLP 2025.

[g] Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting, arXiv 2024.

[h] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, ICML 2024.

[i] StreamForest: Efficient Online Video Understanding with Persistent Event Memory, NeurIPS 2025.


Thank you again for your valuable suggestions. We believe the additional comparisons and ablation studies have strengthened our evaluation, and we would greatly appreciate your reconsideration of our work.

Reviewer_X4wUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StreamBank is an inference framework for streaming video understanding that combines generation acceleration with bounded event memory. It uses shallow-layer multi-token prediction heads to propose tokens in parallel and optionally reuses previously verified responses when adjacent turns are sufficiently similar. A frozen target model verifies all candidates, preserving greedy decoding equivalence under a fixed context. The framework also uses draft–verifier representation differences to guide event promotion and eviction. Experiments with a Qwen2.5-VL-3B-based MMDuet2 target report modest improvements on streaming benchmarks, stronger gains on several long-video benchmarks, and a 2.01× decoding-throughput speedup on ProactiveVideoQA.

Critical strengths
  1. A coherent connection between generation acceleration and memory management. StreamBank reuses target-verification feedback to guide event retention, linking two aspects of streaming inference that are usually optimized separately. This provides a useful research direction toward prediction-aware memory management.
  2. Verified decoding preserves correctness under a fixed context. Parallel MTP proposals and reused responses both undergo target-model verification. The paper provides an explicit algorithm and a prefix-invariant argument establishing greedy equivalence under the same context, clearly separating decoding correctness from the effects of memory compression.
  3. Encouraging results across streaming and long-video benchmarks. With a 3B target model, StreamBank reports a 2.01× decoding-throughput speedup on ProactiveVideoQA while slightly improving overall pAUC. It also improves the reproduced baseline on OVO-Bench and StreamingBench, with larger gains on LongVideoBench, MLVU, and LVBench, suggesting that bounded event memory can reduce redundant context without necessarily sacrificing answer quality.
Critical weaknesses

The central memory-utility signal is insufficiently validated. StreamBank interprets draft–verifier representation differences as evidence of historical-event utility. However, disagreement can arise from draft-model limitations or prediction difficulty without indicating that an event contributed to the correction. Equations (9)–(11) establish representational alignment rather than an event’s contribution to prediction. The paper provides neither counterfactual evidence-removal tests nor component ablations isolating verifier gain and correction evidence. Consequently, the main conceptual contribution remains insufficiently supported.

The availability of verification feedback under realistic streaming protocols is unclear. Algorithm 1 records event feedback at rejected MTP positions during response generation. It is therefore unclear how informative feedback accumulates during long silent periods, before the first retrospective question, or under short multiple-choice outputs. This matters because event retention must occur before future questions are known. The paper does not sufficiently explain the causal update schedule or evaluate whether previously unsupported events survive until they become relevant.

The experiments do not isolate the benefits of the proposed generation–memory coupling. The system combines parallel MTP, response reuse, event summaries, novelty, direct-response support, and verification-derived scores, but the ablations largely examine reuse, representative-frame budgets, and basic retention policies. The EPM gain over matched-budget FIFO is only 1.35 points, without uncertainty estimates. These comparisons do not establish whether verification feedback provides a meaningful advantage over a simpler memory policy combined with an existing speculative decoder.

The efficiency evidence is too narrow to establish the overall system benefit. The reported 2.01× improvement measures decoding throughput on ProactiveVideoQA using one 3B target model and one A800 GPU. The paper does not provide absolute throughput, comprehensive response latency, or a breakdown including prefill, event-caption generation, and memory maintenance. It also does not clearly isolate decoding acceleration from the effect of shorter memory contexts. These omissions limit the practical interpretation of the principal efficiency result.

The baseline coverage is inadequate for both claimed contributions. The acceleration evaluation lacks comparisons with established speculative decoding methods, including the cited Medusa/EAGLE-style approaches and SpecVLM, while CVPR 2026 ParallelVLM is not discussed. The memory evaluation is limited to FIFO, visual similarity, and full history, rather than stronger streaming-memory alternatives. Without controlled comparisons on the same target model and resource budget, it is difficult to assess the benefit of the proposed framework over existing decoding and memory components.

Questions for authors

What evidence shows that the proposed alignment scores measure event utility rather than draft-model error? Could you compare these scores with the change in target-model likelihood or answer correctness when individual events are removed? Please also evaluate whether changing draft quality substantially changes event retention despite keeping the target model fixed.

How does event memory receive feedback before a question or response occurs? Please specify the update schedule during silent periods and single-query evaluation. For OVO-Bench and StreamingBench, what rejected MTP positions are available to update memory before answering the evaluated question? Are any auxiliary generations performed, and are their costs included?

Which components are responsible for the quality improvements? Could you separately ablate verifier gain, correction evidence, direct-response support, novelty, and semantic captions while holding memory capacity and decoding settings fixed? A comparison between the complete policy and a version without speculation-derived scores would directly test the claimed coupling.

How much end-to-end acceleration remains after including all system overhead? Please report absolute tokens/s, TTFT, complete response latency, and total streaming computation, including event-caption generation and memory maintenance. Can you compare AR, MTP, and MTP-plus-reuse under exactly the same retained context and output settings?

How does StreamBank compare with stronger decoding and memory baselines? Could you compare applicable implementations of SpecVLM and ParallelVLM, alongside a Medusa/EAGLE-style baseline, on the same target model? For memory, could you include a stronger streaming-memory policy and report both quality and resource use under matched budgets?

Can you clarify the implementation details and the differing VAD speedup results? Please provide the reuse thresholds, reused-prefix length, event-segmentation and representative-frame selection rules, and optimizer/training settings. Table 2 reports a 2.26× VAD speedup, whereas Figure 3 and its accompanying text report 2.40× with the default two-frame budget; what configuration or measurement difference explains this discrepancy?

4 comments
AuthorsOpenReview ↗

Answer to Reviewer X4wU [part1/3]

We sincerely thank you for recognizing the merits of our work, particularly the coherent connection between generation acceleration and memory management, the generation correctness ensured by target-model verification, and the encouraging results on streaming and long-video benchmarks. We address your concerns below.

[W1 & Q1] **Insufficient validation of memory-utility signals. **

Thank you for this important suggestion. We agree that it is essential to distinguish genuine event utility from draft-model prediction errors. To address this concern, we conducted two additional experiments investigating the contribution of selected events to target-model predictions and the sensitivity of event retention to draft quality.

(1) Counterfactual event removal. On the EPM subset of OVO-Bench, we remove either the highest-scoring event or a randomly selected event before final-answer generation, while keeping the target model unchanged.

ConditionAccuracy (%)
No removal52.5
Remove highest49.8
Remove random51.2

Removing the highest-scoring event reduces accuracy by 2.7 percentage points, compared with 1.3 points for random removal. The larger degradation suggests that the event-promotion scores identify historical events that contribute more to the target model's answer correctness.

(2) Sensitivity to draft checkpoints. We evaluate three draft checkpoints on EGO while keeping the target model and memory configuration fixed.

DraftMTP acceptanceBank JaccardpAUC
epoch00148.2%0.9327.5
epoch00964.8%0.9627.6
epoch01870.0%1.0027.7

Despite substantial differences in draft quality (MTP acceptance of 48.2%–70.0%), the resulting event banks remain highly consistent (Jaccard ≥ 0.93 relative to epoch018), with only a 0.2-point variation in pAUC. Together, these experiments suggest that verification-derived scores capture useful historical information and that event retention is robust to changes in draft-model quality, rather than being primarily driven by draft-model errors.

[W2 & Q2] Unclear availability and update mechanism of verification feedback in realistic streaming scenarios.

Thank you for highlighting this important issue. We apologize for not clearly explaining how verification feedback is obtained under streaming evaluation in the original submission. For OVO-Bench and StreamingBench, we process video frames sequentially as multi-turn inputs and generate a short caption at each turn. These captions undergo MTP drafting and target-model verification, providing feedback for subsequent memory updates without accessing future questions.

We acknowledge that this procedure introduces additional computation. However, when an event is promoted to memory, its previously generated caption can be reused as the event description, avoiding redundant generation. OVO-Bench and StreamingBench report accuracy rather than decoding speedup; ProactiveVideoQA requires no auxiliary caption generation, so its reported speedup is unaffected by this overhead. When no valid verification feedback is available, StreamBank relies on visual continuity, visual novelty, and previously accumulated support for memory management. We have clarified the feedback mechanism and evaluation procedure in Section 4.1.

[W3 & Q3] Insufficient ablation of generation–memory coupling.

Thank you for this constructive suggestion. We agree that isolating individual components is important for validating generation–memory coupling. To directly examine the role of speculation-derived feedback, we additionally ablate Verifier Gain and Correction Evidence on ProactiveVideoQA, assessing their individual and combined contributions.

FeedbackOverall pAUC
Neither29.5
Verifier Gain only30.7
Correction Evidence only30.8
Full feedback31.0

Removing both signals reduces overall pAUC from 31.0 to 29.5. Using only Verifier Gain or Correction Evidence yields 30.7 and 30.8, respectively, while combining them achieves the best result (31.0). This demonstrates that both verification-derived signals contribute to effective event retention and provide complementary information. More importantly, the 1.5-point improvement over the policy without speculation-derived feedback provides direct evidence for the benefit of generation–memory coupling.

AuthorsOpenReview ↗

Answer to Reviewer X4wU [part2/3]

[W4 & Q4] Insufficient end-to-end efficiency evaluation.

Thank you for this valuable suggestion. We agree that decoding throughput alone is insufficient to characterize practical streaming efficiency. We therefore add a comparison of AR, MTP, and MTP + Reuse, reporting end-to-end speedup, time-to-first-token (TTFT), and complete response latency.

MethodE2E SpeedupTTFT (ms)Latency (ms)
MMDuet21.00×225.7 / 1203.81033.8 / 3757.5
MTP2.71×108.7 / 320.4899.7 / 3361.3
MTP + Reuse2.85×108.1 / 317.2896.6 / 2986.3

E2E Speedup is computed as the ratio of total processing time under the same workload, including video processing, memory updates, event-caption generation, and response generation. TTFT and response latency measure per-response performance and are reported as P50/P95.

MTP achieves a 2.71× speedup over MMDuet2, increasing to 2.85× with response reuse. MTP + Reuse reduces P50/P95 TTFT from 225.7/1203.8 ms to 108.1/317.2 ms, and complete response latency from 1033.8/3757.5 ms to 896.6/2986.3 ms. These results demonstrate improvements in both generation efficiency and initial and overall response latency.

[W5 & Q5] Insufficient comparisons with stronger decoding and memory baselines.

Thank you for your valuable suggestion. We agree that stronger decoding and memory baselines are important for a comprehensive assessment of StreamBank. We therefore conduct two additional comparisons.

(1) Comparison with existing speculative decoding methods. On the VAD subset of ProactiveVideoQA, we compare StreamBank with SpecVLM [a], Kangaroo [b], and Medusa [c] under the same frozen target model. For SpecVLM, we adapt its implementation using Qwen2.5-0.5B as the draft model. We also considered ParallelVLM [d], but its official implementation code has not been released, preventing reliable reproduction under our experimental setup. We therefore exclude it from the quantitative comparison and discuss it in Related Work.

MethodDecode Speedup
Target AR1.00×
SpecVLM†1.22×
Kangaroo1.51×
Medusa2.05×
StreamBank (untrained)0.63×
StreamBank (w/o reuse)1.84×
StreamBank2.40×

† Our adaptation of SpecVLM using Qwen2.5-0.5B as the draft model.

StreamBank achieves a 2.40× decoding speedup, outperforming SpecVLM (1.22×), Kangaroo (1.51×), and Medusa (2.05×). Without response reuse, StreamBank reaches 1.84×, below Medusa. Both methods use multi-head token prediction, while Medusa additionally employs tree-based verification to explore multiple candidate paths, potentially improving token acceptance. StreamBank instead gains an additional advantage in streaming video by reusing target-verified responses or prefixes across adjacent turns, raising its speedup to 2.40×. The untrained StreamBank variant achieves only 0.63×, indicating the importance of dedicated MTP training.

(2) Comparison with stronger memory-retention baselines. We adapt a StreamForest-style[e] retention policy to our fixed-capacity event bank while keeping the event representations and memory budget unchanged.

Retention policyEPM accuracy (%)
Recent only50.2
FIFO51.2
Visual similarity49.8
Full history48.8
StreamForest-style51.5
StreamBank52.5

On OVO-Bench EPM, StreamBank achieves 52.53% accuracy, outperforming StreamForest-style (51.52%) and FIFO (51.18%). These results further support the effectiveness of verification-guided retention in preserving useful historical information under a fixed memory budget.

AuthorsOpenReview ↗

Answer to Reviewer X4wU [part3/3]

[Q6] Insufficient implementation details and inconsistent VAD speedup results.

Thank you for carefully identifying the insufficient implementation details and inconsistent VAD speedup. We apologize for this reporting discrepancy: the 2.26× speedup in Table 2 was a typographical error; the correct value is 2.40×, consistent with Figure 3. There is no difference in experimental configuration or measurement protocol, and this error does not affect the experimental analyses or conclusions. We will correct it in the revised manuscript. The implementation settings are provided below.

(1) Implementation settings.

ComponentConfiguration
Target modelMMDuet2 (Qwen2.5-VL-3B), frozen
Parallel MTP3 prediction heads branching from layer 4
MTP confidence threshold0.6
Reuse thresholdsτlow=0.75, τhigh=0.92\tau_{\mathrm{low}}=0.75,\ \tau_{\mathrm{high}}=0.92
Reused-prefix lengthKp=4K_p=4 tokens
Recent window32 frames
Event bank capacity16 events
Representative-frame budget2 frames per event
Verification feedback thresholdsδg=δc=0.1, Nmin⁡=2\delta_g=\delta_c=0.1,\ N_{\min}=2
Promotion parameters(αs,αn)=(0.8,0.2), τp=0.75(\alpha_s,\alpha_n)=(0.8,0.2),\ \tau_p=0.75
Retention parameters(βs,βa,βn,βl)=(0.5,0.2,0.2,0.1)(\beta_s,\beta_a,\beta_n,\beta_l)=(0.5,0.2,0.2,0.1)
HardwareSingle NVIDIA A800 GPU

(2) Training settings. We train the MTP modules for 20 epochs using the AdamW optimizer with a learning rate of 1×10−51\times10^{-5} and β=(0.9,0.95)\beta=(0.9,0.95). We use a per-device batch size of 1 and 32 gradient accumulation steps, yielding an effective batch size of 32 on a single GPU. The learning rate follows a linear warmup for 30 optimizer update steps and then linearly decays. We set the maximum sequence length to 10,000 tokens, apply gradient clipping with a maximum norm of 1.0, and use BF16 mixed-precision training.

(3) Event segmentation and representative-frame selection. As frames leave the recent window, we group them into temporally coherent events based on visual continuity. Within each event, duplicate frames are removed while temporal order is preserved; under the default two-frame budget, we retain its first and last frames. For a one-frame event, the single frame is retained. Larger budgets additionally sample frames uniformly along the event timeline while preserving both endpoints.

(4) Cross-turn response reuse. We calculate the cosine similarity sts_t between adjacent-turn visual representations. If st≥0.92s_t\geq0.92, the entire previous response is proposed as a reusable candidate. If 0.75≤st<0.920.75\leq s_t<0.92, only its first four tokens are proposed; otherwise reuse is disabled. Reuse is allowed only when the question is unchanged and the candidate's first token matches the target model's prefill prediction. All remaining candidate tokens undergo target-model verification, and only verified tokens are committed.

[a] SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning, EMNLP 2025.

[b] Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting, arXiv 2024.

[c] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, ICML 2024.

[d] ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding, CVPR 2026.

[e] StreamForest: Efficient Online Video Understanding with Persistent Event Memory, NeurIPS 2025.


Thank you again for your thoughtful and constructive feedback. We hope our additional experiments and clarifications have addressed your concerns, and we would greatly appreciate your reconsideration of our work.

We sincerely thank you for your insightful feedback and constructive suggestions, which have helped us strengthen our paper. We would like to briefly summarize the additional experimental evidence and clarifications provided in our rebuttal to address your concerns.

(1) Verification-Guided Event Memory (W1/W3). We conducted counterfactual event-removal experiments, showing that removing the highest-scoring event causes a larger accuracy drop than random removal (2.7 vs. 1.3 points). Event retention also remains stable across different draft checkpoints (Jaccard ≥ 0.93). Furthermore, removing both Verifier Gain and Correction Evidence reduces pAUC from 31.0 to 29.5, providing evidence for the effectiveness of verification-guided memory management.

(2) Streaming Feedback Mechanism (W2). We clarified how verification feedback is obtained through sequential caption generation during streaming evaluation, including the associated computational overhead and the fallback mechanism when feedback is unavailable.

(3) Efficiency and Stronger Baselines (W4/W5). We added end-to-end efficiency measurements, demonstrating a 2.85× E2E speedup and substantial reductions in TTFT. Under the same target model, StreamBank achieves a 2.40× decoding speedup, outperforming SpecVLM, Kangaroo, and Medusa. It also outperforms FIFO and StreamForest-style memory policies under the same memory budget.

We have additionally clarified the implementation details and corrected the reporting inconsistency identified in Q6.

We believe that these additional experiments and clarifications have adequately addressed your concerns. We would be grateful to know if you have any remaining questions or concerns that we could further clarify. If you find our responses satisfactory, we would sincerely appreciate your reconsideration of our work and its overall assessment.

Thank you again for your time and thoughtful evaluation of our work.

Comments on the paper

1 comment

We sincerely thank all reviewers for their insightful feedback and constructive suggestions, which have helped us strengthen our paper. We summarize our core contributions and the additional experimental evidence provided during the rebuttal.

Our contributions were acknowledged by the reviewers. They recognized the coherent connection between generation acceleration and memory management through shared verification feedback (X4wU, hVoJ), and the decoding correctness guaranteed by target-model verification (X4wU). Reviewers also appreciated the well-motivated framework, clear presentation, and evaluations across streaming and long-video benchmarks (NyYt), as well as the promising improvements in decoding efficiency and historical-event retention (X4wU, hVoJ).

We summarize the main concerns and our responses below:

1. A Unified Framework for Generation Acceleration and Memory Management: Our primary contribution is connecting speculative decoding and historical-event retention through shared target-model verification feedback. StreamBank combines parallel Multi-Token Prediction (MTP) and cross-turn response reuse to exploit within-turn parallelism and temporal redundancy for faster generation. Meanwhile, Verifier Gain and Correction Evidence transform verification feedback into event-utility signals for memory retention. Unlike approaches that optimize these components independently, StreamBank enables generation and memory management to benefit from the same verification process, establishing a unified framework for efficient streaming video understanding.

2. Comprehensive Validation of Verification-Guided Memory Management: We conduct additional experiments to validate the effectiveness of verification-derived feedback. Removing both Verifier Gain and Correction Evidence reduces pAUC from 31.0 to 29.5. Counterfactual event removal shows that removing the highest-scoring event causes a larger accuracy drop than random removal (2.7 vs. 1.3 percentage points), while event retention remains stable across draft-model checkpoints (Jaccard ≥ 0.93). On OVO-Bench EPM, StreamBank achieves 52.53% accuracy, outperforming FIFO (51.18%) and StreamForest-style retention (51.52%). We also expand comparisons with recent streaming video methods, including SimpleStream, Streamo, ThinkStream, VST, and HERMES. These results support the effectiveness of verification-guided memory selection beyond conventional retention policies.

3. Extensive Evaluation of Decoding and End-to-End Efficiency: On the VAD subset of ProactiveVideoQA, StreamBank achieves 2.40× decoding speedup, outperforming SpecVLM (1.22×), Kangaroo (1.51×), and Medusa (2.05×) under the same frozen target model. Cross-turn response reuse further improves speedup from 1.84× to 2.40× over MTP alone. Including video processing, memory updates, event-caption generation, and response generation, StreamBank achieves 2.85× end-to-end speedup over MMDuet2, reducing median time-to-first-token from 225.7 ms to 108.1 ms and response latency from 1033.8 ms to 896.6 ms. These results demonstrate practical efficiency gains beyond decoding throughput.

We have addressed the reviewers' feedback through additional experiments and methodological clarifications. We are confident that these results have addressed the reviewers' primary concerns and strengthened the evidence for our contributions. All updates have been incorporated into the revised manuscript. We sincerely thank the reviewers again for their valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.