acceptodds
Under review as a conference paper at ICLR 2027

AutoStream: Automatic Context Expansion for Streaming Video Understanding

Abstract

Online video understanding requires models to perceive ongoing events while retaining relevant information from the past. Existing training-free methods, however, struggle to balance these two demands: the accumulated memory maintained by history-centric approaches may interfere with real-time perception, whereas recent-window approaches preserve real-time responsiveness at the cost of discarding long-term evidence. We present AutoStream, a training-free framework that realizes a real-time perception first, history on demand principle through automatic context expansion. During stream ingestion, Query-Ready Multi-Context Cache (QRMC) prepares hierarchical caches that span real-time perception and comprehensive historical context. Novelty-Weighted Reservoir Sampling (NWRS) retains temporally dispersed and informative historical evidence, while time-adaptive capacity reallocation progressively shifts the context budget toward long-range coverage as the stream grows. At query time, Confidence-Guided Context Expansion (CGCE) progressively exposes broader history based on model confidence, while its History Shortcut directly activates the comprehensive history for queries identified as history-dependent. By avoiding extra retrieval and auxiliary models, AutoStream maintains low query-time latency. With Qwen2.5-VL-7B as the backbone, AutoStream improves performance from 73.3% to 81.0% on StreamingBench and from 44.7% to 58.2% on the Backward subset of OVO-Bench. With Qwen3-VL-8B, it achieves 67.8% on OVO-Bench and 82.3% on StreamingBench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLMs, Video Understanding, Online Video UnderStanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eketOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AutoStream, a training-free framework that prepares several bounded KV-cache views during stream ingestion and expands query context only when needed. Query-Ready Multi-Context Cache maintains an independently encoded recent cache, a history-conditioned recent cache, and a novelty-weighted reservoir of historical KV states under a 12K-token budget. Confidence-Guided Context Expansion starts from recent context and moves to broader history when the first output-token probability is below a threshold, while a History Shortcut directly selects the full-history level for questions judged to require historical evidence. The method is evaluated on StreamingBench, OVO-Bench, Video-MME, MVBench, and MLVU with several Qwen backbones.

Critical strengths
  1. The "real-time perception first, history on demand" design addresses a real failure mode of streaming systems: indiscriminately adding history can damage current perception. The three cache levels and selective expansion are simple, training-free, and compatible with multiple backbones.

  2. The empirical study is broad. AutoStream is tested on online and offline tasks, includes component and capacity-schedule ablations, and reports matched-backbone comparisons. The results consistently show improved backward-looking performance while largely preserving recent perception.

Critical weaknesses
  1. All reported TTFT values start after the visual caches have already been built. During ingestion, however, each window is processed to update the historical cache and to create both an independent high-resolution recent cache and a history-conditioned high-resolution recent cache. This entails multiple expensive prefills per window, yet the paper reports no ingestion throughput, update latency, real-time factor, or compute relative to SimpleStream/HERMES. LiveStar is a particularly relevant omitted comparison because it combines streaming KV-cache reuse with Peak-End memory compression and reports an online inference speedup. Its proactive objective differs from AutoStream's query-triggered expansion, but comparing the cache construction and memory-maintenance costs would help establish whether AutoStream's three query-ready views provide a favorable systems trade-off. A 35--80 ms query-time TTFT therefore does not establish that AutoStream can construct its "query-ready" states in real time, and the central efficiency claim remains unsupported.

  2. The routing policy is selected and justified on the same benchmarks without showing that it identifies when history is actually useful. First-token probability is only shown to correlate with answer correctness; the appendix explicitly states that it is neither calibrated nor evidence of the causal benefit of expansion. The History Shortcut is another prompt to the same model, but no routing accuracy, oracle-history label comparison, false-shortcut rate, or overhead/accuracy trade-off is reported. The threshold (0.7), duration breakpoints (120/1000 s), and cache allocations are apparently chosen from the evaluation suites. Thus, gains may reflect benchmark-specific routing and repeated inference rather than a generally valid context-selection rule.

Questions for authors
  1. What are the mean/P95 ingestion latency, real-time factor, GPU utilization, and energy/compute per video second when all QRMC prefills and NWRS updates are included? Please compare them with matched SimpleStream, HERMES, and always-full-history baselines.

  2. On an oracle labeling of whether additional history changes a wrong answer to a correct one, what are the precision and recall of first-token thresholding and History Shortcut? How often does expansion change a correct answer to an incorrect one?

  3. Were the confidence threshold, schedule breakpoints, and three token allocations selected on a disjoint validation set? Please report held-out transfer or leave-one-benchmark-out selection.

4 comments

We would like to express our sincere gratitude for your review. Your suggestions will greatly help us improve our paper.

W1 & Q1: Computational Efficiency Analysis of Stream Ingestion

Thank you for this valuable suggestion—LiveStar is a highly relevant comparison, and we will incorporate a discussion of it into our Related Work section.

We benchmark stream ingestion on a single NVIDIA H200 GPU by feeding a 256-second video at 1 FPS. For our method, we report measurements separately across the three stages of Time-Adaptive Capacity Reallocation. All implementations use a unified Qwen2.5-VL backbone.

MethodMean ingestion latency (ms/frame)P95 ingestion latency (ms/frame)Real-time factorGPU memory usage (GB)TFLOPs (per frame)GPU power (W)
SimpleStream-4f192.13193.840.192116.9416.16286.85
HERMES522.06541.650.522118.4629.80249.73
LiveStar366.051422.220.366119.5859.16393.68
Always-full-history588.09940.660.588131.1144.73367.83
Ours, (768,6K)(768, 6\mathrm{K})213.14214.260.213118.1027.31337.53
Ours, (512,8K)(512, 8\mathrm{K})202.07208.550.202118.5519.18297.40
Ours, (256,10K)(256, 10\mathrm{K})196.20197.080.196218.9911.37259.07
  • Because LiveStar generates captions to retain textual memory in addition to visual information, its P95 ingestion latency is notably high, reflecting the latency incurred when generating captions.
  • Always-full-history retains all historical KV states; both its latency and GPU memory usage grow as the input stream lengthens, resulting in the lowest computational efficiency.
  • Similar to our method, baselines such as SimpleStream and HERMES also process high-resolution inputs during ingestion, and prefilling these high-resolution tokens dominates the computational cost.
  • Compared with SimpleStream, our method introduces only an additional low-resolution branch, so neither compute nor latency increases substantially. Compared with HERMES, our method does not compute extra attention scores for KV compression. Meanwhile, cache maintenance (including NWRS) runs very fast, taking less than 50 ms per frame.

W2(1) & Q2: Further Analysis of Routing Policy

We construct oracle labels by evaluating each question separately using Level 1 only, Level 2 only, and Level 3 only. A question answered incorrectly at Level 1 that becomes correct after incorporating additional history (at Level 2 or Level 3) is labeled as an oracle positive.

(1) Precision and recall against oracle labels.

TriggerBenchmarkTrigger rateBase ratePrecisionRecallLift (Precision / Base rate)
First-token thresholdingOVO-Bench36.1%11.0%21.5%70.8%1.96
First-token thresholdingStreamingBench22.2%8.5%23.9%62.6%2.82
History ShortcutOVO-Bench45.2%11.0%13.6%55.9%1.24
History ShortcutStreamingBench21.5%8.5%14.0%35.6%1.65
  • Here, the base rate denotes the proportion of oracle positives across all questions, corresponding to the expected precision of triggering context expansion at random.
  • Compared with random triggering, first-token thresholding achieves 2.0×2.0\times and 2.8×2.8\times the precision on the two benchmarks, along with high recall of 70.8% and 62.6%, respectively. History Shortcut achieves 1.24×1.24\times and 1.65×1.65\times the random-trigger precision and serves as a complementary path to first-token thresholding.
  • These results demonstrate that our context expansion policy effectively identifies samples lacking sufficient context and expands them accordingly.

(2) Frequency of expansion turning a correct answer into an incorrect one.

Benchmark#Q#ExpandedWrong→CorrectCorrect→WrongNet / all questionsCorrect→Wrong / all questions
OVO-Bench14689208278+0.27 pp5.31%
├ Real-Time8373981943−2.87 pp5.14%
└ Backward6315226335+4.44 pp5.55%
StreamingBench249492913767+2.81 pp2.69%
  • The proportion of questions where context expansion changes a correct answer to an incorrect one is 5.31% on OVO-Bench and 2.69% on StreamingBench.
  • Context expansion yields positive net gains on both StreamingBench and the Backward subset of OVO-Bench. Although expanding historical context slightly lowers Real-Time performance on OVO-Bench, the overall average still improves.

(3) Overhead–accuracy trade-off and selective vs. indiscriminate expansion.

  • History Shortcut trade-off. History Shortcut incurs only a lightweight text-only forward pass (~20 ms, already included in the 35–80 ms query-time TTFT reported in the paper), while bringing consistent accuracy improvements on OVO-Bench (65.0→67.165.0 \rightarrow 67.1), StreamingBench (80.4→81.080.4 \rightarrow 81.0), and offline long-video benchmarks (Table 6).
  • Selective routing vs. full-budget / repeated inference. Indiscriminately exposing the full historical budget to every query (Always-Level-3) causes severe accuracy drops—from 67.167.1 to 53.953.9 on OVO-Bench and from 81.081.0 to 70.170.1 on StreamingBench—confirming that our gains stem from selectively choosing the appropriate context per query rather than from simply using more visual tokens or repeated inference.

W2(2) & Q3: Parameter Selection and Held-Out Transfer

(1) Confidence threshold (θ\theta).

We selected θ\theta based on Qwen2.5-VL-7B results on OVO-Bench and StreamingBench (Table 5), and directly transferred this setting to Qwen3-VL backbones and all other benchmarks.

In addition, we report a held-out transfer experiment on RIVER [1], an online video understanding benchmark not involved in selecting θ\theta, evaluating θ∈[0.5,0.8]\theta \in [0.5, 0.8] with Qwen2.5-VL-7B. Here, Live-Perception and Retro-Memory correspond to real-time perception and backward memory, respectively.

Threshold (θ\theta)Live-PerceptionRetro-MemoryAvg.Δ\Delta vs. Default
0.554.3%47.0%48.5%−1.2 pp
0.655.9%46.6%48.5%−1.2 pp
0.7 (Ours)54.8%48.4%49.7%—
0.855.9%47.8%49.4%−0.3 pp

The results show that θ=0.7\theta = 0.7 achieves the highest overall accuracy on this unseen benchmark as well, demonstrating the generalizability of the chosen threshold.

(2) Schedule breakpoints and token allocations.

Both the schedule breakpoints and the three token allocations are key hyperparameters of Time-Adaptive Capacity Reallocation. We designed this reallocation schedule intuitively to balance varying historical information demands across online and offline video understanding, without conducting an exhaustive hyperparameter search.

We further provide held-out transfer experiments on LongVideoBench and LVBench—neither of which was used in parameter selection—retaining the exact stage boundaries (120 s,1000 s)(120\text{ s}, 1000\text{ s}) and (rs,Bs)(r_s, B_s) allocations from the main paper:

Benchmark(256,10K)(256, 10\mathrm{K})(512,8K)(512, 8\mathrm{K})(768,6K)(768, 6\mathrm{K})Ours
LongVideoBench56.8%56.5%56.5%57.1%
LVBench39.5%38.9%38.8%41.8%

Through adaptive reallocation, Time-Adaptive Capacity Reallocation outperforms all three fixed-quota configurations, confirming its generalizability to unseen benchmarks.

We hope these additional efficiency benchmarks, oracle routing analyses, and held-out transfer experiments address your concerns. If you have any further questions, we would be glad to discuss them.


References

[1] Shi Y, Zhao Q, Jiang T, et al. RIVER: A Real-Time Interaction Benchmark for Video LLMs. arXiv preprint arXiv:2603.03985, 2026.

Thank you for the detailed rebuttal. The additional ingestion profiling, oracle-based routing analysis, and held-out transfer experiments address several of my concerns and substantially strengthen the paper. However, the modest routing precision and cases where context expansion harms real-time perception leave some uncertainty about the reliability of the routing policy. While my assessment is now more positive, the remaining concerns prevent me from raising the score. I therefore maintain my original rating of 2.

Reviewer_uBcfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AutoStream, a training-free framework for online video understanding realizing “real-time perception first, history on demand” principle. It introduces Query-Ready Multi-Context Cache (QRMC), which maintains recent and historical KV caches using novelty-weighted reservoir sampling and time-adaptive capacity reallocation. At query time, Confidence-Guided Context Expansion (CGCE) progressively expands the available context based on first-token probability, while a History Shortcut directly activates historical context for history-dependent queries. Experiments across several Qwen-VL backbones on online and offline video understanding benchmarks demonstrate improvements in historical reasoning while maintaining competitive real-time perception at low query-time latency.

Critical strengths

The paper is very well-written, with clear explanations, strong motivation, and a coherent flow throughout. The extensive comparisons against recent baselines, comprehensive ablation and inclusion of latency measurements are appreciated.

Critical weaknesses
  • W1. [Marginal improvement over baselines] Despite the well-motivated framework, the improvements over prior baselines are relatively marginal and inconsistent across backbones. In particular, AutoStream does not consistently improve both real-time perception and historical understanding compared to existing on-demand approaches such as OASIS and StreamScout. Instead, it often improves one at the expense of the other, without consistently advancing the perception-memory Pareto frontier over existing approaches.

    • W1.1. Interestingly, Qwen3-VL-4B slightly outperforms Qwen3-VL-8B on the overall OVO-Bench score when including HLD, contrary to the general scaling trend. Given that confidence-guided context expansion relies on a fixed first-token probability threshold across backbones, it is unclear whether differences in confidence calibration and subsequent expansion decisions contribute to this discrepancy. A backbone-wise analysis of confidence distributions and their relationship to accuracy would help clarify this behavior. Additionally, evaluating a different model family, such as InternVL3.5 or a GLM-V variant, would provide stronger evidence of generalizability beyond the Qwen-VL family.
    • W1.2. For Tab. 2, could the authors clarify the input settings for each approach, given that questions in these offline benchmarks are typically posed after the entire video has been observed? In particular, does SimpleStream only retain the final four frames of each video? Furthermore, although AutoStream is designed to preserve historical information, its performance remains below the original Qwen2.5-VL-7B backbone across all reported offline benchmarks. Could the authors explain this degradation and clarify the frame-sampling strategies and visual-token budgets of all the baselines? Finally, since Tab. 2 is part of the main result, and we have already observed from Tab. 1 that AutoStream gains/performance is not consistent across backbones, extending the offline evaluation to Qwen3-VL-4B/8B is crucial to understand its effectiveness.
  • W2. [Ablations] The individual benefits of some proposed components remain limited. In Tab. 7, NWRS yields only marginal improvements over random reservoir sampling, while the time-adaptive capacity reallocation in Tab. 11 offers limited gains over simpler fixed-budget configurations. This raises questions about whether their additional complexity is sufficiently justified. Furthermore, Tab. 6 suggests that History Shortcut contributes substantially to performance, particularly on history-dependent benchmarks. An ablation comparing shortcut-only routing against the full expansion mechanism would help establish the necessity of CGCE.

    • W2.1. For Tab. 4, could the authors report separate results for real-time perception and backward tracing to clarify where each context level contributes? Additionally, does the StreamingBench average refer only to real-time visual understanding or to all task categories?
Questions for authors

Q1. [Streaming Efficiency] While the query-time efficiency analysis in Tab. 3 is appreciated, it excludes the continuous overhead of visual encoding, KV construction and maintenance. The latter is important for quantifying latency in end-to-end streaming settings. Could the authors quantify these costs.

I particularly appreciate the motivation, clarity, and completeness of the paper. However, the empirical results do not fully match the strength of its presentation and motivation. I am therefore recommending a Weak Reject, but remain open to increasing my rating based on the rebuttal and after reading the other reviews.

6 comments

We sincerely thank you for taking the time to review our paper. We greatly appreciate your recognition of the motivation, clarity, and completeness of our work.

W1: Perception–Memory Trade-Off

Our method achieves consistent improvements on both the Real-Time Visual Understanding subset of StreamingBench and the Backward subset of OVO-Bench.

We understand that your concern also stems from the noticeable drop relative to the recent-only SimpleStream baseline on the Real-Time subset of OVO-Bench when using Qwen2.5-VL-7B. This occurs because the threshold θ\theta governs the trade-off between historical context and real-time perception: our default setting θ=0.7\theta = 0.7 favors incorporating historical context, whereas lowering θ\theta recovers Real-Time accuracy:

Methodθ\thetaOVO-Bench (Real-Time)OVO-Bench (Backward)
Qwen2.5-VL-7B + SimpleStream—78.451.9
Qwen2.5-VL-7B + Ours0.775.958.2
Qwen2.5-VL-7B + Ours0.378.056.2

At θ=0.3\theta = 0.3, Real-Time accuracy closely approaches SimpleStream while retaining a substantial improvement on Backward tasks despite a modest decrease relative to θ=0.7\theta = 0.7.

W1.1: Backbone Scaling, Confidence Calibration, and Cross-Family Generalization

(1) Confidence calibration and Qwen3-VL-4B vs. 8B scaling analysis.

  • Similar mean confidence masks different distributions. Mean final-answer first-token probabilities (0.929 vs. 0.925) and ECE (0.236 vs. 0.241; 10 equal-width bins) are close. However, 4B produces more high confidence predictions (p≥0.99p \ge 0.99), both overall (60.4% vs. 56.2%) and on Backward tasks (48.5% vs. 42.5%). The table reports mean confidence and accuracy high-confidence p; all paired values are 4B / 8B.

The table reports mean confidence, accuracy, the proportion of high-confidence predictions (p≥0.99p \ge 0.99), the number of correct high-confidence predictions, and Expected Calibration Error (ECE) for each subtask; all paired values are 4B / 8B, and lower ECE is better.

Backward SubtaskMean ConfidenceAccuracy (%)High-confidence predictions (%)Correct count at high-confidenceECE
EPM0.911 / 0.89557.6 / 51.548.1 / 43.8105 / 970.343 / 0.381
HLD0.883 / 0.88244.6 / 40.939.2 / 31.738 / 250.443 / 0.475
ASI0.933 / 0.91664.9 / 72.360.8 / 53.466 / 660.286 / 0.194
  • Confidence–accuracy relationships differ across backbones. Across the three Backward subtasks, Qwen3-VL-4B produces high-confidence predictions (p≥0.99p \ge 0.99) more frequently than Qwen3-VL-8B. It yields more correct high-confidence predictions on EPM and HLD, while the two models have the same number on ASI. The ECE results indicate better confidence calibration for 4B on EPM and HLD, whereas 8B is better calibrated on ASI.

  • Subtask weighting reverses the ranking when HLD is excluded. OVO-Bench equally weights Real-Time and Backward, giving each Backward subtask 1/61/6 of the overall score. 4B's EPM (+1.0) and HLD (+0.6) contributions offset 8B's ASI (−1.2) and Real-Time (−0.1) advantages, yielding +0.3 overall. Excluding HLD increases the weights of EPM and ASI to 1/41/4, and 8B leads on both Backward (61.9 vs. 61.2) and overall score (71.2 vs. 70.8, +0.4).

  • In the revised manuscript, we will include the full per-subtask results of our method and all baselines on both OVO-Bench and StreamingBench so that readers can clearly trace the sources of performance gains.

(2) Evaluation on non-Qwen backbone families.

We further evaluate our method on two additional backbone families—GLM-4.6V-Flash and InternVL3.5-4B-Instruct—to demonstrate cross-family generalizability:

MethodOVO-BenchStreamingBench
GLM-4.6V-Flash56.374.4
GLM-4.6V-Flash + SimpleStream66.579.2
GLM-4.6V-Flash + Ours68.180.8
InternVL3.5-4B-Instruct55.776.8
InternVL3.5-4B-Instruct + SimpleStream56.977.7
InternVL3.5-4B-Instruct + Ours58.578.9

W1.2: Offline Evaluation Protocol, Visual Budgets, and Qwen3-VL Results

(1) Offline evaluation protocol for backbones.

For the offline evaluation of backbones, we use the following input settings:

BenchmarkFPSFrame limitPer-frame max tokens
MVBench1—768
MLVU1102464
Video-MME1102464

Because videos in MVBench are relatively short, we impose no frame limit and allow up to 768 tokens per frame. For MLVU and Video-MME, we sample at 1 FPS with at most 1024 frames and up to 64 tokens per frame.

(2) Sampling and visual-token budgets for all compared methods.

On offline benchmarks, all streaming methods (SimpleStream, DSCache, OASIS, and AutoStream) ingest the entire video sequentially as a stream and answer questions posed at the end of the video:

  • SimpleStream (official setting): Yes, SimpleStream retains the final 4 frames of each video, as it takes 1 FPS streaming input and retains only the latest 4 frames in a sliding window (~768 tokens per frame, ~3.1K tokens in total).
  • DSCache (official setting): 1 FPS streaming input; the instant cache retains the latest 5 frames (256 tokens for each of the 4 older frames and ~1024 tokens for the most recent frame), and the history cache retains up to 28 frames, for a total budget of ~9.2K tokens.
  • OASIS (official setting): 2 FPS input for the recent window (~16 frames) and 1 FPS input for memory construction, with a 32-frame short-term memory; when long-term memory retrieval is triggered, at most 2 groups of up to 16 frames each are retrieved. Each frame uses 256 tokens with adjacent-frame merging, yielding a total budget of ~10K tokens.
  • AutoStream: evaluated directly under the exact streaming settings described in the paper without any modification—1 FPS streaming input under a bounded 12K total visual-token budget (with the 4-frame recent window adaptively set to 768/512/256 tokens per frame and the historical cache set to 6K/8K/10K tokens at 64 tokens per frame).

(3) Comparison with the backbone.

  • The backbone operates without streaming constraints and can preserve historical information much more completely.
  • Unlike history-centric memory construction methods, AutoStream is designed specifically for streaming scenarios: it balances real-time perception and memory within a bounded budget rather than aiming to retain the full video history.
  • Compared with peer streaming methods such as OASIS, our adaptive historical context expansion strategy outperforms retrieval-based recall and achieves higher accuracy.

(4) Offline benchmark results on Qwen3-VL-4B / 8B.

Following the exact same evaluation protocol, we compare the backbones, SimpleStream, and our method on Qwen3-VL-4B and Qwen3-VL-8B:

MethodMVBenchMLVUVideo-MME (Long)Video-MME (All)
Qwen2.5-VL-7B65.067.953.863.3
Qwen3-VL-4B63.870.760.067.2
Qwen3-VL-8B65.173.660.468.9
Qwen2.5-VL-7B + SimpleStream56.347.638.341.9
Qwen3-VL-4B + SimpleStream56.447.439.241.3
Qwen3-VL-8B + SimpleStream56.346.541.243.1
Qwen2.5-VL-7B + Ours64.166.253.662.0
Qwen3-VL-4B + Ours63.161.151.858.5
Qwen3-VL-8B + Ours65.765.654.762.3
  • Under our method, Qwen3-VL-8B consistently outperforms Qwen3-VL-4B across all reported metrics, demonstrating positive scaling behavior under our streaming evaluation protocol.
  • Although the Qwen3-VL backbones surpass Qwen2.5-VL-7B on MLVU and Video-MME, these offline advantages do not uniformly carry over to SimpleStream or our method.
  • As evidenced by SimpleStream, a stronger backbone on offline benchmarks does not automatically translate into higher streaming performance under a recent-window constraint. Building upon a recent-first design with on-demand history, our method delivers substantial, consistent gains over SimpleStream across every backbone; nevertheless, how to more fully preserve a backbone's offline strengths under streaming recent-window constraints remains a promising direction for future exploration.

W2: Component Necessity and Ablations

(1) Necessity of NWRS and Time-Adaptive Capacity Reallocation.

  • Compared with Random-only and Novelty-only, NWRS achieves better overall performance across benchmarks. Meanwhile, under frame-by-frame ingestion, novelty scoring and sorting add less than 5 ms of overhead, and total cache maintenance takes under 50 ms per frame without introducing substantial extra computation.
  • Time-Adaptive Capacity Reallocation reconciles the competing demands for historical coverage and real-time perception across online and offline video understanding benchmarks, while generalizing effectively to unseen benchmarks without retuning:
Benchmark(256,10K)(256, 10\mathrm{K})(512,8K)(512, 8\mathrm{K})(768,6K)(768, 6\mathrm{K})Ours
LongVideoBench56.856.556.557.1
LVBench39.538.938.841.8

(2) Comparison between Shortcut-only and the full expansion mechanism.

We further compare Shortcut only, CGCE only, and the full combination:

PolicyOVO RTOVO BWOVO Avg.StreamingBenchMVBenchMLVUVideo-MME
SimpleStream78.451.965.278.556.347.641.9
CGCE only75.554.665.080.461.059.853.7
Shortcut only77.954.866.480.263.563.560.2
Shortcut + CGCE (Ours)75.958.267.181.064.166.262.0
  • Role of History Shortcut and CGCE. As expected, History Shortcut contributes substantially on history-intensive tasks (especially offline long-video benchmarks) by routing explicitly history-dependent questions directly to expanded context, avoiding overconfident wrong answers on insufficient recent frames. Meanwhile, CGCE only already yields notable gains over SimpleStream across both online and offline benchmarks (e.g., +12.2 on MLVU and +11.8 on Video-MME).
  • Shortcut only is insufficient. Relying solely on Shortcut only consistently degrades accuracy across all benchmarks compared with the full method: OVO-Bench Backward drops by 3.4 points (58.2 →\rightarrow 54.8, with overall Avg. −0.7) and StreamingBench drops by 0.8 points, while MVBench, MLVU, and Video-MME drop by 0.6, 2.7, and 1.8 points, respectively. Because question text alone cannot catch every query whose recent frames happen to lack key visual evidence, CGCE provides essential confidence-based fallback expansion, making the two components complementary.

W2.1: Table 4 Breakdown and StreamingBench Evaluation Scope

(1) Breakdown of Table 4.

We supplement Table 4 with separate OVO-Bench Real-Time and Backward results when ablating each context level:

SplitReal-TimeBackwardHLDBackward (w/o HLD)
w/o Level 173.150.431.759.7
w/o Level 274.649.929.660.0
w/o Level 376.457.755.458.9
Ours75.958.254.360.2
  • Removing Level 1 causes the largest drop in Real-Time performance, followed by removing Level 2, whereas removing Level 3 has the smallest impact on Real-Time perception.
  • HLD is not a well-defined backward-tracing task: when historical information is almost entirely absent (w/o Level 3), the model’s inability to answer can also lead to artificially inflated HLD performance. We therefore additionally report Backward (w/o HLD), which shows that removing Level 3 has the largest impact on the average performance across the other two backward-tracing subtasks (EPM and ASI).

(2) StreamingBench evaluation scope.

The reported StreamingBench average covers only the Real-Time Visual Understanding subset; we will clarify this explicitly in the revised manuscript.

Reviewer_HKbPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers training-free online video understanding and observes that history-centric memory methods let accumulated context interfere with perception of the current moment, whereas recent-window methods such as SimpleStream remain accurate on real-time questions but lose long-range evidence. AutoStream keeps three caches during ingestion: a high-resolution cache of the latest four frames prefilled without history, a second high-resolution cache of the same frames prefilled after the historical cache, and a historical cache of low-resolution frame blocks whose membership is decided by weighted reservoir sampling with novelty weights (NWRS); a fixed 12K-token budget is shifted from recent resolution (768, 512, 256 tokens per frame) to historical capacity (6K, 8K, 10K tokens) at 120 s and 1000 s. When a question arrives, the method answers from the recent-only context and moves to the history-conditioned recent context and then to the full historical context whenever the first-token probability is below θ = 0.7, while a History Shortcut, obtained by prompting the same MLLM, sends questions judged to need history directly to the last level. On Qwen2.5-VL-7B the method raises StreamingBench from 73.3% to 81.0% and OVO-Bench Backward from 44.7% to 58.2%, with Qwen3-VL-8B it reaches 82.3% on StreamingBench and 67.8% on OVO-Bench, and on offline benchmarks it recovers most of the long-video accuracy lost by SimpleStream (Video-MME 41.9 to 62.0). The authors claim that exposing history per query, rather than applying one context policy to all queries, reconciles real-time perception with long-term memory at low query-time cost.

Critical strengths
  1. This work proposes to prepare every candidate context before the question arrives and to choose among them at query time from the model's own confidence. Since the ladder only requires additional first-token predictions over caches that already exist, there is no retrieval, re-encoding or auxiliary embedding model at query time, and TTFT stays below 80 ms at 1024 frames (Table 10). The expansion rule is also independent of the memory design: added to HERMES, it raises OVO-Bench real-time accuracy from 69.0% to 74.8% with Backward unchanged or slightly improved (Table 9).

  2. This work introduces a historical cache policy that does not depend on knowing the stream length. Using the exponential-key form of weighted reservoir sampling (Efraimidis and Spirakis, 2006) with weights bounded in [1, β], it keeps a stochastic coverage of the whole stream while biasing retention toward frames that are novel locally or relative to the current reservoir, and Appendix D states correctly how this reduces to uniform reservoir sampling for β = 1. The time-adaptive reallocation of the recent and historical budgets is a reasonable complement, and Table 11 shows that no single fixed allocation is best across all six metrics.

  3. This work evaluates the method across five backbones and on both online and offline benchmarks, and it tests the assumption behind the confidence trigger. Gains over SimpleStream on StreamingBench and OVO-Bench Backward hold for Qwen2.5-VL 7B and 32B and Qwen3-VL 4B, 8B and 30B-A3B, and the correlation between first-token probability and correctness is examined with video-clustered robust standard errors and Holm correction across eight tests (Appendix B), with the limits of that analysis stated explicitly.

Critical weaknesses
  1. Lack of any measurement of ingestion cost, which leaves the efficiency claim incomplete. For every window, the method prefills the low-resolution blocks of the previous window conditioned on the historical cache, prefills the current window at high resolution without history, and prefills it again at high resolution conditioned on a historical cache of 6K to 10K tokens (Eqs. 1 and 2). The per-window compute is therefore several times that of SimpleStream, and three caches plus the frame embeddings used for novelty scoring must be held in memory, but the efficiency analysis in Section 4.4 starts only when the question arrives. Per-frame ingestion latency, sustainable frame rate on one GPU, and memory during ingestion, compared with SimpleStream, DSCache and HERMES on the same hardware, are needed to support the statement that the method maintains high computational efficiency.

  2. Lack of evidence that the confidence trigger works outside multiple-choice evaluation, and lack of routing statistics. In every benchmark used, the first output token is an option letter, so first-token probability is a direct measure of answer confidence; for open-ended or proactive questions the first token is usually uninformative. The threshold θ = 0.7 is selected from test-set results (Table 5), and Level 3 is always taken as the final answer when the first two levels fail, even when its confidence is lower than that of Level 1 (the example in Figure 2 changes its answer at every level). The History Shortcut is itself a query-time classification by the MLLM, and removing it costs 6.4 points on MLVU and 8.3 on Video-MME (Table 6), so much of the offline gain may come from routing questions to Level 3 rather than from confidence-based expansion. A supplementary experiment should report the fraction of questions resolved at each level and the shortcut trigger rate per benchmark and task type, the number of answers that change from correct to wrong after expansion, and an open-ended streaming QA setting with a confidence measure that does not rely on option letters.

  3. Lack of budget-matched baselines, together with a real-time result that contradicts the stated principle. AutoStream uses a 12K-token context while SimpleStream uses four recent frames, so part of the improvement may be due to more visual tokens. Despite the "real-time perception first" principle, OVO-Bench real-time accuracy is lower than SimpleStream on every backbone where both are reported: 78.4 to 75.9 on Qwen2.5-VL-7B, 81.4 to 80.6 on Qwen3-VL-8B, 83.0 to 79.6 on Qwen2.5-VL-32B. DSCache also scores higher on StreamingBench with the same backbone (82.3 versus 81.0), and AutoStream remains below the unmodified backbone offline (Video-MME 62.0 versus 63.3, MVBench 64.1 versus 65.0). A supplementary experiment with SimpleStream given the same 12K budget (more recent frames or higher resolution), an "always Level 3" policy and an oracle that picks the best level per question would show how much of the gain comes from adaptive expansion itself.

  4. Lack of a description and analysis of positional and causal consistency in the concatenated caches. Each historical block is prefilled while attending to the reservoir as it was at insertion time, and some of those attended blocks are later evicted; the retained blocks are then reordered by time and concatenated with a recent cache that was prefilled against yet another reservoir state. Level 2 goes further and uses the history-conditioned recent cache without the history it attended to, so its keys and values encode attention to tokens that are absent from the context. The paper does not say how MRoPE positions are assigned after eviction (gaps preserved or indices compacted) or how large the resulting staleness is. Since Levels 2 and 3 depend on these caches, a comparison with re-prefilling the retained frames from scratch is needed.

  5. Lack of statistical support for the individual components, whose novelty is also limited. Compared with FIFO, NWRS improves OVO-Bench by 0.3 and StreamingBench by 0.5 points; Random-only is slightly better than NWRS on OVO-Bench (67.2 versus 67.1), and Novelty-only equals it on Video-MME Long (53.6). Because NWRS draws a random key per block, results over several seeds are required to interpret differences of this size, yet single runs are reported throughout. The stage boundaries at 120 s and 1000 s and the budgets are set by hand, and the constant (256, 10K) allocation is better on MLVU and the Video-MME average (Table 11). The components themselves adapt existing ideas: weighted reservoir sampling (Efraimidis and Spirakis, 2006), separately built recent and historical caches (DSCache, Pang et al., 2026), in-context KV cache retrieval (ReKV, Di et al., 2025) and the use of the confidence of an initial answer to decide whether further computation is needed (VideoAuto-R1, Liu et al., 2026, which the paper cites as its inspiration). The contribution therefore rests on the combination, and the effect of each part should be shown to exceed run-to-run variation.

Questions for authors
  1. What are the per-frame ingestion latency, the sustainable frame rate and the peak memory during ingestion, compared with SimpleStream and HERMES on the same GPU?
  2. What fraction of questions ends at each level, how often does the History Shortcut fire, and how many answers change from correct to wrong after expansion, per benchmark and task type?
  3. How are positions assigned to retained historical blocks after eviction, and does re-prefilling the retained frames change accuracy?
  4. How does SimpleStream perform with the same 12K-token budget, and how does an "always Level 3" policy perform?
  5. What is the variance across random seeds for NWRS?
  6. How was θ = 0.7 chosen, and was any data other than the reported test benchmarks used for this choice?
  7. Why does real-time accuracy on OVO-Bench decrease relative to SimpleStream if Level 1 reproduces the recent-only context?
5 comments

We sincerely thank you for your thoughtful and constructive comments, which have been invaluable in helping us improve the quality of our work.

W1 & Q1: Ingestion Cost

We analyze the computational efficiency during the stream ingestion stage on a single NVIDIA H200 GPU with 1 FPS video input, under both frame-by-frame (chunk=1) and 4-frame window (chunk=4) settings (latency reported in ms/frame).

Because Time-Adaptive Capacity Reallocation uses different input budgets across stages, we report measurements separately for each of the three stages. All implementations use a unified Qwen2.5-VL backbone.

Methodchunk=1 latency (ms/frame)chunk=1 sustainable FPSchunk=1 memory (GB)chunk=4 latency (ms/frame)chunk=4 sustainable FPSchunk=4 memory (GB)
SimpleStream-4f192.135.2016.94236.694.2217.41
DSCache565.451.7720.07188.085.3220.16
HERMES522.061.9218.46224.624.4518.74
Ours, (768,6K)(768, 6\mathrm{K})213.144.6918.10118.198.4618.62
Ours, (512,8K)(512, 8\mathrm{K})202.074.9518.5594.3010.6018.80
Ours, (256,10K)(256, 10\mathrm{K})196.205.1018.9973.6513.5819.13
  • Similar to our method, baselines such as SimpleStream, DSCache and HERMES also process high-resolution inputs during ingestion, and prefilling these high-resolution tokens dominates the computational cost.
  • Compared with SimpleStream, our method introduces only an additional low-resolution branch, so neither compute nor latency increases substantially.
  • Unlike HERMES, which computes extra attention scores for KV compression, and DSCache, which repeatedly re-prefills historical content beyond the recent window, our method avoids both overheads. Meanwhile, cache maintenance (including NWRS) is lightweight, taking less than 50 ms per frame.

W2(1) & Q2: Routing Statistics and Expansion Reliability

(1) Shortcut trigger rate and final-level distribution per benchmark.

We collect per-question routing logs from the Qwen2.5-VL-7B runs reported in our main tables:

Benchmark#QShortcut rate#CGCEEnd at L1End at L2End at L3
OVO-Bench146845.2%80437.3%48.8%13.9%
├ Real-Time83727.1%61052.4%31.1%16.5%
└ Backward63169.3%19417.3%72.3%10.5%
StreamingBench249421.5%195462.8%3.8%33.5%
MVBench400068.4%86613.1%12.6%74.4%
MLVU217481.7%3979.1%1.5%89.4%
Video-MME270091.4%2334.0%0.5%95.5%
  • Consistent with question semantics, Real-Time questions tend to terminate at Level 1 or Level 2, whereas Backward questions and offline benchmarks predominantly terminate at Level 2 or Level 3.
  • On OVO-Bench, History Shortcut routes triggered questions to Level 2 to better handle HLD tasks, whereas on all other benchmarks it routes them to Level 3. We will clarify it in the revised manuscript.

(2) Relationship between Shortcut trigger rate and task type.

The trigger rates of History Shortcut align closely with question semantics across task categories:

  • StreamingBench: Counting (92.2%), Causal Reasoning (64.8%), Prospective Reasoning (63.0%), and Event Understanding (31.0%); in contrast, Object, Action, Attribute, and Spatial Understanding trigger the shortcut only 3.7%–4.4% of the time.
  • OVO-Bench: Backward-tracing tasks reach 98.0% (ASI), 63.4% (HLD), and 58.6% (EPM), whereas real-time tasks (ACR, ATR, OJR, STU) range from 12.8% to 25.3%.
  • Video-MME: Short / Medium / Long videos exhibit trigger rates of 82.9% / 92.8% / 98.4%, respectively.

(3) Frequency of expansion turning a correct answer into an incorrect one.

Benchmark#Q#ExpandedWrong→CorrectCorrect→WrongNet / all questionsCorrect→Wrong / all questions
OVO-Bench14689208278+0.27 pp5.31%
├ Real-Time8373981943−2.87 pp5.14%
└ Backward6315226335+4.44 pp5.55%
StreamingBench249492913767+2.81 pp2.69%
  • The proportion of questions where context expansion flips a correct answer to an incorrect one is 5.31% on OVO-Bench and 2.69% on StreamingBench.
  • Context expansion yields positive net gains on both StreamingBench and the Backward subset of OVO-Bench. Although expanding historical context reduces Real-Time accuracy on OVO-Bench, the overall average still improves.

W2(2): Final-Level Selection (Always-Level-3 vs. Max-Confidence)

We conduct an additional experiment using a max-confidence fallback—selecting the level with the highest confidence when no level meets the threshold requirement—instead of defaulting to Level 3 at the final step:

MethodOVO-BenchStreamingBench
Max-confidence fallback66.680.0
Ours67.181.0
  • The results show that the max-confidence fallback leads to a drop in accuracy. Confidence scores obtained under different context levels are not directly comparable; thus, confidence is better suited for threshold-based filtering at each level rather than cross-level maximum-confidence selection.
  • Failing to pass the threshold at the first two levels indicates that recent context is insufficient, making Level 3—which contains the most complete historical information—the appropriate final choice.

W2(3): Contribution of History Shortcut vs. CGCE

We further compare Shortcut only, CGCE only, and the full combination:

PolicyOVO RTOVO BWOVO Avg.StreamingBenchMVBenchMLVUVideo-MME
SimpleStream78.451.965.278.556.347.641.9
CGCE only75.554.665.080.461.059.853.7
Shortcut only77.954.866.480.263.563.560.2
Shortcut + CGCE (Ours)75.958.267.181.064.166.262.0
  • Role of History Shortcut and CGCE. As expected, History Shortcut contributes substantially on history-intensive tasks (especially offline long-video benchmarks) by routing explicitly history-dependent questions directly to expanded context, avoiding overconfident wrong answers on insufficient recent frames. Meanwhile, CGCE only already yields notable gains over SimpleStream across both online and offline benchmarks (e.g., +12.2 on MLVU and +11.8 on Video-MME).
  • Shortcut only is insufficient. Relying solely on Shortcut only consistently degrades accuracy across all benchmarks compared with the full method: OVO-Bench Backward drops by 3.4 points (58.2 →\rightarrow 54.8, with overall Avg. −0.7) and StreamingBench drops by 0.8 points, while MVBench, MLVU, and Video-MME drop by 0.6, 2.7, and 1.8 points, respectively. Because question text alone cannot catch every query whose recent frames happen to lack key visual evidence, CGCE provides essential confidence-based fallback expansion, making the two components complementary.

W2(4) & Q6: Selection of θ\theta and Use of Held-out Data

We selected θ\theta based on Qwen2.5-VL-7B performance on OVO-Bench and StreamingBench (Table 5), and directly transferred this setting to Qwen3-VL models and all other benchmarks.

In addition, we report a held-out transfer experiment on RIVER [1], an online video understanding benchmark not used in selecting θ\theta, evaluating θ∈[0.5,0.8]\theta \in [0.5, 0.8] with Qwen2.5-VL-7B. Here, Live-Perception and Retro-Memory correspond to real-time perception and backward memory, respectively.

Threshold (θ\theta)Live-Perception (372 Qs)Retro-Memory (1488 Qs)Overall Acc.Δ\Delta vs. Default
0.554.3%47.0%48.5%−1.2 pp
0.655.9%46.6%48.5%−1.2 pp
0.7 (Ours)54.8%48.4%49.7%—
0.855.9%47.8%49.4%−0.3 pp

The results confirm that θ=0.7\theta = 0.7 also achieves the highest overall accuracy on this unseen benchmark, demonstrating the generalizability of our chosen θ\theta.


References

[1] Shi Y, Zhao Q, Jiang T, et al. RIVER: A Real-Time Interaction Benchmark for Video LLMs. arXiv preprint arXiv:2603.03985, 2026.

W2(5): Open-ended Streaming QA

To examine whether token probabilities still carry signal about answer correctness in open-ended responses, we convert OVO-Bench and StreamingBench into open-ended streaming QA settings by removing candidate answer options, using an LLM judge to evaluate answer correctness. We then compare three model confidence proxies: first-token probability, mean probability of the first 8 generated tokens, and mean probability across all generated tokens.

We report both the ROC-AUC for distinguishing correct from incorrect answers and the QA accuracy:

Confidence CriterionOVO-OE Correctness AUCStreamingBench-OE Correctness AUCOVO-OE Avg.StreamingBench-OE Avg.
First token0.5620.54337.247.3
Mean prob. of first 8 tokens0.6590.64337.748.5
Mean prob. of all tokens0.6750.66637.948.3
  • For open-ended questions, first-token probability alone is indeed weak at reflecting model confidence (AUC of 0.562 / 0.543). In contrast, without relying on option letters, the mean probability of a short 8-token prefix substantially improves correctness discrimination (AUC of 0.659 / 0.643, approaching full-response AUC) and translates into consistent end-to-end accuracy gains over first-token routing on both OVO-OE (37.2 →\rightarrow 37.7) and StreamingBench-OE (47.3 →\rightarrow 48.5), closely matching full-token routing. -Moreover, 93% of responses exceed 8 tokens, indicating that this prefix confidence signal can generally be obtained before the full answer is generated. Using the 8-token prefix as model confidence yields an average effective TTFT of ~300 ms, which readily meets streaming QA responsiveness requirements.

W3(1) & Q4: Results of Budget-matched SimpleStream, Always-Level-3, and Oracle

(1) Budget-matched SimpleStream.

Using the same 12K total visual-token budget without capping per-frame tokens, we evaluate SimpleStream variants across different backbones and recent-window sizes:

OVO-Bench:

Model4 frames (official)4 frames8 frames16 frames32 frames
Qwen2.5-VL-7B65.261.661.361.259.8
Qwen3-VL-4B65.866.165.666.265.9
Qwen3-VL-8B67.767.767.567.668.1

StreamingBench:

Model4 frames (official)4 frames8 frames16 frames32 frames
Qwen2.5-VL-7B78.579.579.779.877.8
Qwen3-VL-4B80.379.480.881.680.9
Qwen3-VL-8B80.680.181.081.180.6

Simply increasing the total visual-token budget or enlarging the recent window does not bring consistent accuracy gains, and a clear gap remains compared with our method. Moreover, on the Qwen2.5-VL backbone, feeding overly large per-frame visual token counts can even degrade performance.

(2) Results of the Always-Level-3 variant.

We additionally report results for the Always-Level-3 policy (in Backward and Avg., values before and after / denote scores with and without HLD, respectively):

MethodReal-TimeBackwardAvg.StreamingBench
Ours75.958.2 / 60.267.1 / 68.181.0
Always-Level-361.046.8 / 58.053.9 / 59.570.1
  • Using Level 3 exclusively causes a substantial drop in real-time performance across both benchmarks.
  • Because HLD does not faithfully reflect backward-tracing performance, we also report results excluding HLD (w/o HLD). Unlike real-time accuracy, Backward (w/o HLD) performance is largely maintained under Always-Level-3.

(3) Oracle results and the contribution of adaptive expansion.

We further report the performance of each fixed single-level policy alongside an Oracle policy that selects the best level per question using ground-truth labels:

MethodReal-TimeBackwardAvg.StreamingBench
Always-Level-178.451.965.278.5
Always-Level-275.557.666.579.3
Always-Level-361.046.853.970.1
Ours75.958.267.181.0
Oracle84.271.878.087.5

Comparing fixed single-level policies, our method, and the Oracle demonstrates that adaptive context expansion allows our method to surpass every individual fixed level and move closer to Oracle accuracy, validating the effectiveness of adaptive expansion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.