- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AutoStream, a training-free framework that prepares several bounded KV-cache views during stream ingestion and expands query context only when needed. Query-Ready Multi-Context Cache maintains an independently encoded recent cache, a history-conditioned recent cache, and a novelty-weighted reservoir of historical KV states under a 12K-token budget. Confidence-Guided Context Expansion starts from recent context and moves to broader history when the first output-token probability is below a threshold, while a History Shortcut directly selects the full-history level for questions judged to require historical evidence. The method is evaluated on StreamingBench, OVO-Bench, Video-MME, MVBench, and MLVU with several Qwen backbones.
Critical strengths
-
The "real-time perception first, history on demand" design addresses a real failure mode of streaming systems: indiscriminately adding history can damage current perception. The three cache levels and selective expansion are simple, training-free, and compatible with multiple backbones.
-
The empirical study is broad. AutoStream is tested on online and offline tasks, includes component and capacity-schedule ablations, and reports matched-backbone comparisons. The results consistently show improved backward-looking performance while largely preserving recent perception.
Critical weaknesses
-
All reported TTFT values start after the visual caches have already been built. During ingestion, however, each window is processed to update the historical cache and to create both an independent high-resolution recent cache and a history-conditioned high-resolution recent cache. This entails multiple expensive prefills per window, yet the paper reports no ingestion throughput, update latency, real-time factor, or compute relative to SimpleStream/HERMES. LiveStar is a particularly relevant omitted comparison because it combines streaming KV-cache reuse with Peak-End memory compression and reports an online inference speedup. Its proactive objective differs from AutoStream's query-triggered expansion, but comparing the cache construction and memory-maintenance costs would help establish whether AutoStream's three query-ready views provide a favorable systems trade-off. A 35--80 ms query-time TTFT therefore does not establish that AutoStream can construct its "query-ready" states in real time, and the central efficiency claim remains unsupported.
-
The routing policy is selected and justified on the same benchmarks without showing that it identifies when history is actually useful. First-token probability is only shown to correlate with answer correctness; the appendix explicitly states that it is neither calibrated nor evidence of the causal benefit of expansion. The History Shortcut is another prompt to the same model, but no routing accuracy, oracle-history label comparison, false-shortcut rate, or overhead/accuracy trade-off is reported. The threshold (0.7), duration breakpoints (120/1000 s), and cache allocations are apparently chosen from the evaluation suites. Thus, gains may reflect benchmark-specific routing and repeated inference rather than a generally valid context-selection rule.
Questions for authors
-
What are the mean/P95 ingestion latency, real-time factor, GPU utilization, and energy/compute per video second when all QRMC prefills and NWRS updates are included? Please compare them with matched SimpleStream, HERMES, and always-full-history baselines.
-
On an oracle labeling of whether additional history changes a wrong answer to a correct one, what are the precision and recall of first-token thresholding and History Shortcut? How often does expansion change a correct answer to an incorrect one?
-
Were the confidence threshold, schedule breakpoints, and three token allocations selected on a disjoint validation set? Please report held-out transfer or leave-one-benchmark-out selection.
4 comments
W2(1) & Q2: Further Analysis of Routing Policy
We construct oracle labels by evaluating each question separately using Level 1 only, Level 2 only, and Level 3 only. A question answered incorrectly at Level 1 that becomes correct after incorporating additional history (at Level 2 or Level 3) is labeled as an oracle positive.
(1) Precision and recall against oracle labels.
| Trigger | Benchmark | Trigger rate | Base rate | Precision | Recall | Lift (Precision / Base rate) |
|---|---|---|---|---|---|---|
| First-token thresholding | OVO-Bench | 36.1% | 11.0% | 21.5% | 70.8% | 1.96 |
| First-token thresholding | StreamingBench | 22.2% | 8.5% | 23.9% | 62.6% | 2.82 |
| History Shortcut | OVO-Bench | 45.2% | 11.0% | 13.6% | 55.9% | 1.24 |
| History Shortcut | StreamingBench | 21.5% | 8.5% | 14.0% | 35.6% | 1.65 |
- Here, the base rate denotes the proportion of oracle positives across all questions, corresponding to the expected precision of triggering context expansion at random.
- Compared with random triggering, first-token thresholding achieves and the precision on the two benchmarks, along with high recall of 70.8% and 62.6%, respectively. History Shortcut achieves and the random-trigger precision and serves as a complementary path to first-token thresholding.
- These results demonstrate that our context expansion policy effectively identifies samples lacking sufficient context and expands them accordingly.
(2) Frequency of expansion turning a correct answer into an incorrect one.
| Benchmark | #Q | #Expanded | Wrong→Correct | Correct→Wrong | Net / all questions | Correct→Wrong / all questions |
|---|---|---|---|---|---|---|
| OVO-Bench | 1468 | 920 | 82 | 78 | +0.27 pp | 5.31% |
| ├ Real-Time | 837 | 398 | 19 | 43 | −2.87 pp | 5.14% |
| └ Backward | 631 | 522 | 63 | 35 | +4.44 pp | 5.55% |
| StreamingBench | 2494 | 929 | 137 | 67 | +2.81 pp | 2.69% |
- The proportion of questions where context expansion changes a correct answer to an incorrect one is 5.31% on OVO-Bench and 2.69% on StreamingBench.
- Context expansion yields positive net gains on both StreamingBench and the Backward subset of OVO-Bench. Although expanding historical context slightly lowers Real-Time performance on OVO-Bench, the overall average still improves.
(3) Overhead–accuracy trade-off and selective vs. indiscriminate expansion.
- History Shortcut trade-off. History Shortcut incurs only a lightweight text-only forward pass (~20 ms, already included in the 35–80 ms query-time TTFT reported in the paper), while bringing consistent accuracy improvements on OVO-Bench (), StreamingBench (), and offline long-video benchmarks (Table 6).
- Selective routing vs. full-budget / repeated inference. Indiscriminately exposing the full historical budget to every query (Always-Level-3) causes severe accuracy drops—from to on OVO-Bench and from to on StreamingBench—confirming that our gains stem from selectively choosing the appropriate context per query rather than from simply using more visual tokens or repeated inference.
W2(2) & Q3: Parameter Selection and Held-Out Transfer
(1) Confidence threshold ().
We selected based on Qwen2.5-VL-7B results on OVO-Bench and StreamingBench (Table 5), and directly transferred this setting to Qwen3-VL backbones and all other benchmarks.
In addition, we report a held-out transfer experiment on RIVER [1], an online video understanding benchmark not involved in selecting , evaluating with Qwen2.5-VL-7B. Here, Live-Perception and Retro-Memory correspond to real-time perception and backward memory, respectively.
| Threshold () | Live-Perception | Retro-Memory | Avg. | vs. Default |
|---|---|---|---|---|
| 0.5 | 54.3% | 47.0% | 48.5% | −1.2 pp |
| 0.6 | 55.9% | 46.6% | 48.5% | −1.2 pp |
| 0.7 (Ours) | 54.8% | 48.4% | 49.7% | — |
| 0.8 | 55.9% | 47.8% | 49.4% | −0.3 pp |
The results show that achieves the highest overall accuracy on this unseen benchmark as well, demonstrating the generalizability of the chosen threshold.
(2) Schedule breakpoints and token allocations.
Both the schedule breakpoints and the three token allocations are key hyperparameters of Time-Adaptive Capacity Reallocation. We designed this reallocation schedule intuitively to balance varying historical information demands across online and offline video understanding, without conducting an exhaustive hyperparameter search.
We further provide held-out transfer experiments on LongVideoBench and LVBench—neither of which was used in parameter selection—retaining the exact stage boundaries and allocations from the main paper:
| Benchmark | Ours | |||
|---|---|---|---|---|
| LongVideoBench | 56.8% | 56.5% | 56.5% | 57.1% |
| LVBench | 39.5% | 38.9% | 38.8% | 41.8% |
Through adaptive reallocation, Time-Adaptive Capacity Reallocation outperforms all three fixed-quota configurations, confirming its generalizability to unseen benchmarks.
We hope these additional efficiency benchmarks, oracle routing analyses, and held-out transfer experiments address your concerns. If you have any further questions, we would be glad to discuss them.
References
[1] Shi Y, Zhao Q, Jiang T, et al. RIVER: A Real-Time Interaction Benchmark for Video LLMs. arXiv preprint arXiv:2603.03985, 2026.
Thank you for the detailed rebuttal. The additional ingestion profiling, oracle-based routing analysis, and held-out transfer experiments address several of my concerns and substantially strengthen the paper. However, the modest routing precision and cases where context expansion harms real-time perception leave some uncertainty about the reliability of the routing policy. While my assessment is now more positive, the remaining concerns prevent me from raising the score. I therefore maintain my original rating of 2.
We would like to express our sincere gratitude for your review. Your suggestions will greatly help us improve our paper.
Thank you for this valuable suggestion—LiveStar is a highly relevant comparison, and we will incorporate a discussion of it into our Related Work section.
We benchmark stream ingestion on a single NVIDIA H200 GPU by feeding a 256-second video at 1 FPS. For our method, we report measurements separately across the three stages of Time-Adaptive Capacity Reallocation. All implementations use a unified Qwen2.5-VL backbone.