acceptodds
Under review as a conference paper at ICLR 2027

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Abstract

Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual-processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Streaming Video Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7nbSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WRWS, a training-free framework for evidence selection in streaming video understanding. It uses an external encoder and an entropy-based weight to balance semantic relevance with temporal recency, selecting observations before expensive target-model visual encoding.

Critical strengths
  1. The paper is very easy to follow, it addresses a practical problem: balancing historical evidence and recent context under a limited visual-processing budget.
  2. The framework requires no training or access to target-model internal states, making it easy to integrate with different Video-LLMs.
  3. The evaluation covers multiple model families and benchmarks, with additional task-level analyses and qualitative failure cases.
Critical weaknesses
  1. The design is highly heuristic and essentially relies on a simple weighted combination, providing insufficient novelty for ICLR.
  2. The parameter α\alpha experiments in Table 4(a) and Appendix E.1 (Table 14) do not convincingly establish the effectiveness of the weighting mechanism, as the improvements are negligible.
  3. Table 2 shows no clear performance advantage over related methods.
  4. The reported reduction to 47.93% of baseline VTAT is not a meaningful efficiency comparison, since it compares against encoding the entire observed history. SimpleStream could simply buffer the latest four raw observations and encode them only when a query arrives.
Questions for authors
  1. Does the efficiency advantage remain against a Recent-only baseline that buffers the latest four raw observations and encodes them only when a query arrives, under the same visual budget?
  2. Which processing stages are included in VTAT and inference TTFT? Does WRWS also reduce latency measured from query arrival to the first answer token?
7 comments

Response to Reviewer #7nbS (1/2)

W1 — Methodological contributions beyond heuristic weighted scoring

We appreciate the reviewer's concern and would like to clarify that the contribution of WRWS extends beyond the weighted combination itself, focusing on uncertainty-adaptive evidence allocation and the accuracy–efficiency trade-off in streaming video understanding.

First, WRWS adaptively incorporates historical evidence while preserving a strong recency prior under a bounded visual-processing budget. Unlike fixed evidence-selection policies, WRWS uses query-specific retrieval uncertainty to balance semantic relevance and temporal recency.

Second, WRWS performs evidence selection before expensive target-model encoding. Unlike OASIS [1], which relies on preliminary MLLM reasoning, and ShallowStream [2], which uses target-model internal signals, WRWS operates entirely through lightweight external representations without additional training or access to target-model states.

Importantly, efficiency is central to streaming video understanding, where continuous visual inputs and real-time requirements make computational cost particularly critical. WRWS addresses this practical challenge by limiting expensive visual encoding to selected observations while maintaining competitive accuracy, achieving a 52.1% VTAT reduction against the evaluated encode-first baseline.


W2 — Query-adaptive allocation versus fixed-α\alpha policies

The key advantage of WRWS is that it outperforms backbone-specific optimal fixed weightings using a unified uncertainty-adaptive rule, without requiring backbone-specific tuning. As shown in Appendix E.1, Table 14, the best fixed α\alpha differs across backbones: α=1.0\alpha=1.0 for Qwen2.5-VL-7B and α=0.75\alpha=0.75 for Qwen3-VL-8B. In contrast, WRWS applies the same adaptive rule to both backbones and achieves higher accuracy by 0.25 and 0.34 points, respectively.

ModelAllocationBackwardReal-TimeForwardB+RAll
Qwen2.5-VL-7Bα=0.00\alpha=0.00 (Semantic-only)47.1259.0839.1753.1048.46
α=0.25\alpha=0.2547.9764.4640.7256.2151.05
α=0.50\alpha=0.5050.2268.3439.9659.2852.84
α=0.75\alpha=0.7549.9773.9538.5861.9654.16
α=1.00\alpha=1.00 (Recent-only)50.4377.2039.5663.8255.73
Adaptive50.5977.1340.2363.8655.98
Qwen3-VL-8Bα=0.00\alpha=0.00 (Semantic-only)48.4662.8743.9555.6751.76
α=0.25\alpha=0.2548.8069.7245.9059.2654.81
α=0.50\alpha=0.5049.5673.3245.5761.4456.15
α=0.75\alpha=0.7549.9078.3946.4764.1458.25
α=1.00\alpha=1.00 (Recent-only)49.4980.7044.1165.0958.10
Adaptive50.0080.7445.0565.3758.59

W3 — Table 2 shows no clear performance advantage over related methods

We would like to emphasize that WRWS achieves measurable accuracy improvements in several settings, while its primary contribution lies in achieving a favorable accuracy–efficiency trade-off rather than pursuing accuracy gains alone.

First, WRWS achieves competitive or improved accuracy across multiple Video-LLM backbones. As shown in Table 2, WRWS improves over SimpleStream from 80.59 to 81.11 on StreamingBench and from 57.55 to 58.59 on OVO-Bench with Qwen3-VL-8B. Similar improvements are observed on other base model.

ModelStreamingBench Real-TimeOVO-Bench BackwardOVO-Bench Real-TimeOVO-Bench ForwardOVO-Bench Avg.
Qwen3-VL-8B80.5144.0065.0046.3051.77
SimpleStream80.5948.8481.2642.5557.55
WRWS (Ours)81.1150.0080.7445.0558.59
Qwen3-VL-30B-A3B80.7563.2475.4359.1565.94
SimpleStream83.5161.4585.4844.1263.68
WRWS (Ours)83.7162.3584.9445.7764.35

Second, WRWS substantially reduces expensive visual-processing costs while maintaining competitive accuracy. As summarized in the table below, WRWS reduces average VTAT by 52.1%, corresponding to a 2.09× speedup over the evaluated encode-first SimpleStream baseline on Qwen2.5-VL-7B, while improving accuracy on both benchmarks.

Together, these results highlight the practical value of WRWS in achieving substantial computational savings while preserving competitive or improved accuracy, which is particularly important for streaming video understanding under real-time constraints.

MethodStreamingBench Accuracy (%) ↑OVO-Bench Accuracy (%) ↑VTAT (s) ↓Reduction ↑Speedup ↑
SimpleStream78.4755.7312.18—1.00×
WRWS (Ours)79.4755.985.8452.1%2.09×

Response to Reviewer #7nbS (2/2)

W4 & Q1 — Efficiency–capability trade-off between WRWS and buffered Recent-only

We agree that buffering recent raw observations and encoding them only upon query arrival is a valid and computationally efficient alternative. We clarify that the reported 47.93% VTAT ratio is measured against the evaluated encode-first SimpleStream pipeline.

Importantly, the efficiency of buffered Recent-only also depends on the query workload. In practical streaming scenarios, users may issue multiple queries over the same continuous video (e.g., at t=1,3,5,…t=1,3,5,\ldots seconds). While buffering avoids expensive target-VLM encoding before queries arrive, each query still requires encoding the selected observations. Without cross-query feature caching, overlapping observation windows may lead to repeated encoding of the same frames, making cumulative computation dependent on query frequency and reuse strategies.

More importantly, the key distinction lies in the ability to access historical evidence under the same bounded target-model visual-processing budget. Buffered Recent-only selects only the latest KK observations, potentially missing relevant earlier evidence. In contrast, WRWS incrementally maintains lightweight representations of the observed history and selectively retrieves query-relevant observations, while encoding at most KK observations with the target Video-LLM per query.

Thus, WRWS enables history-aware evidence selection with bounded target-model computation, rather than restricting evidence to a fixed recent window. The reported efficiency improvement demonstrates its advantage over the evaluated encode-first pipeline, while the buffered alternative represents a distinct accuracy–efficiency trade-off.


Q2 — Clarifying VTAT&TTFT measurements, and the source of WRWS's efficiency gains

We would like to clarify that VTAT and inference TTFT capture different stages of streaming video understanding. Unlike offline video understanding, streaming systems continuously receive and process incoming video observations before a user query arrives. Accordingly, VTAT measures the elapsed time from the first visual observation to the first answer token, capturing the overall streaming pipeline, whereas inference TTFT starts only when the target Video-LLM begins answer inference, excluding preceding stream processing.

As shown in Appendix C (Table 5), target-VLM visual encoding accounts for 62.36% of the baseline VTAT, motivating our retrieve-first, encode-later design. In contrast, Appendix F (Table 17) shows nearly identical inference TTFT for WRWS and SimpleStream (0.2431 s vs. 0.2429 s), consistent with their identical target-model visual budgets. Thus, WRWS achieves its efficiency gains primarily by reducing cumulative visual-processing costs, rather than accelerating the final answer inference stage. We will clarify this distinction in the revised manuscript.

MetricTarget-VLM visual encodingOther pipeline overheadInference TTFTTotal VTAT
Time (s)7.5964.3420.24312.181
Fraction (%)62.3635.651.99100.00
Group# SamplesWRWS VTAT (s)SimpleStream VTAT (s)VTAT RatioWRWS Inference TTFT (s)SimpleStream Inference TTFT (s)TTFT Ratio
Overall30355.8412.1847.93%0.24310.2429100.08%

[1] Zhijia Liang, Jiaming Li, Weikai Chen, Yanhao Zhang, Haonan Lu, and Guanbin Li. OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning. arXiv preprint arXiv:2604.17052, 2026.

[2] Jitai Hao, Ke Yang, Qiang Huang, and Jun Yu. ShallowStream: Index Shallow Then Answer Deep for Streaming Video Understanding. arXiv preprint arXiv:2609.02780, 2026.

Thank you for the detailed rebuttal and clarifications. I appreciate the authors' efforts in addressing my concerns, particularly the explanations of the efficiency metrics and the practical motivation behind WRWS.

However, my main concerns remain. First, the improvements of adaptive weighting over the best fixed configurations are still marginal (0.25 and 0.34 points), which does not sufficiently demonstrate the effectiveness of the proposed mechanism. Second, the reported efficiency advantage is primarily established against an encode-first baseline, while the requested comparison with a buffered Recent-only baseline remains unanswered. Therefore, the claimed efficiency gains are not yet sufficiently supported by a fair comparison.

Overall, I acknowledge the practical value of this work, but I still find the methodological novelty and empirical advantages insufficiently convincing for ICLR. I will maintain my original score.

Reviewer_KxnfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed WRWS, a training-free method to slove the problem how to balance the recent frame and the retrival historial frames for the VLM to process the Streaming video. By defining the uncertainty, this paper proposal the equation for estimize which frames need to consider. t estimates retrieval uncertainty from the query–observation similarity distribution and uses this uncertainty to adaptively combine semantic relevance with a temporal recency prior for frame selection. But this paper has two main problems. Firstly, the novelty seems limited, because it just calculates the similarity and combines retrieval and the recent frames. Secondly, the experiments and comparisons are not thorough, and the reported gains are limited—no competing method is evaluated consistently across all backbones, no comparison is made against SOTA keyframe-selection methods such as AKS/FOCUS, and the paper's own ablation shows the adaptive allocation improves over a pure-recency baseline by only 0.25/0.34 points.

Critical strengths
  1. The paper addresses a practically important problem. Deferring expensive target-VLM visual encoding until query evidence is known is well motivated for continuous video streams, where processing every incoming observation can be computationally wasteful.
  2. The retrieve-first, encode-later design provides substantial efficiency benefits. WRWS reduces average vision-to-answer time from 12.18 to 5.84 seconds on Qwen2.5-VL-7B while retaining comparable accuracy. The efficiency gain also increases with stream length.
  3. The method is training-free and model-external. WRWS does not require access to target-model hidden states, attention, logits, or historical KV caches. This makes it easy to apply across different Video-LLM families and compatible with optimized serving backends such as vLLM.
Critical weaknesses

Limited methodological novelty. WRWS mainly applies a linear interpolation between normalized semantic relevance and a linear recency prior, with the mixing coefficient derived from normalized entropy. All components are standard, and the resulting method is a simple scoring heuristic. Please clarify the technical novelty beyond combining existing semantic retrieval, recency weighting, and entropy-based uncertainty.

Limited gain from adaptive allocation. The adaptive mechanism improves over Recent-only by only 0.25 points on Qwen2.5-VL and 0.49 points on Qwen3-VL, and produces exactly the same selections as Recent-only for 79.9% of queries. These small gains provide limited evidence for the practical value of the paper’s core mechanism. Please explain when uncertainty adaptation is actually beneficial and provide results on settings where it yields a meaningful improvement over the simple recency baseline.

Questions for authors

Limited methodological novelty. WRWS mainly applies a linear interpolation between normalized semantic relevance and a linear recency prior, with the mixing coefficient derived from normalized entropy. All components are standard, and the resulting method is a simple scoring heuristic. Please clarify the technical novelty beyond combining existing semantic retrieval, recency weighting, and entropy-based uncertainty.

Limited gain from adaptive allocation. The adaptive mechanism improves over Recent-only by only 0.25 points on Qwen2.5-VL and 0.49 points on Qwen3-VL, and produces exactly the same selections as Recent-only for 79.9% of queries. These small gains provide limited evidence for the practical value of the paper’s core mechanism. Please explain when uncertainty adaptation is actually beneficial and provide results on settings where it yields a meaningful improvement over the simple recency baseline.

Inconsistent baseline coverage. Table 2 uses different competitors for different backbone families, while Qwen3.5 is compared only with its base model. Please report common baselines—at least Recent-only, Semantic-only, and the best fixed-α policy—across every backbone under the same sampling rate and frame budget.

Missing keyframe-selection baselines. The paper does not compare against recent training-free selectors such as AKS or FOCUS, despite addressing the same budgeted frame-selection problem. Please evaluate them under the same K-frame budget, restricting them to causally available frames for streaming evaluation, and report both accuracy and selection cost.

5 comments

Response to Reviewer #Kxnf (1/2)

W1 & Q1 — Methodological contributions beyond the standard scoring formulation

We appreciate the reviewer's concern and would like to clarify that WRWS is not an arbitrary combination of standard components, but an empirically motivated framework designed to achieve a favorable accuracy–efficiency trade-off in streaming video understanding.

First, our design is grounded in two empirical observations from Sec. 2. Expensive target-VLM visual encoding motivates evidence selection before target-model processing, while the strong performance of Recent-only and its query-level complementarity with Semantic-only motivate uncertainty-adaptive evidence allocation under a bounded visual-processing budget.

Second, we deliberately favor simple yet effective designs. For instance, Table 4(b) shows comparable performance between linear and exponential recency weighting, achieving 58.59 and 58.31, respectively. We therefore adopt the simpler linear formulation without introducing unnecessary complexity.

Importantly, efficiency is central to streaming video understanding, where continuous visual inputs and real-time requirements make computational cost particularly critical. Unlike OASIS[1] and ShallowStream[2], WRWS performs uncertainty-adaptive evidence selection using lightweight external representations without preliminary MLLM reasoning or access to target-model internal states. This enables historical evidence retrieval with bounded target-model computation, achieving a 52.1% VTAT reduction against the evaluated encode-first baseline while maintaining competitive accuracy.


W2 & Q2 — Interpreting the 79.9% selection overlap and the gains from adaptive allocation

We would like to clarify that the 79.9% selection overlap is consistent with our design motivation, as WRWS aims to preserve the empirically strong recency prior while making targeted evidence adjustments.

To directly evaluate the effectiveness of these adjustments, we conduct an additional analysis, focusing on the queries where WRWS selects different observations from Recent-only. On this subset, WRWS improves accuracy by 2.29 points over Recent-only. Following the same within-task uncertainty stratification protocol as Appendix E.2, Table 16, we observe larger improvements in Q3, Q4, and Q5, reaching 3.62, 5.30, and 4.46 points, respectively, as detailed below.

MethodQ1Q2Q3Q4Q5Overall
# Queries124105138132112611
Recent-only58.0665.7151.4559.0951.7956.96
WRWS (Ours)58.0662.8655.0764.3956.2559.25
Δ\Delta (pp)0.00-2.86+3.62+5.30+4.46+2.29

These results demonstrate that uncertainty-adaptive allocation can yield meaningful improvements on specific queries while preserving the strong recency prior across most observations.


Response to Reviewer #Kxnf(2/2)

Q3 — Cross-backbone baseline comparisons and controlled ablations

We appreciate the reviewer's concern and would like to clarify the distinct purposes of our main comparisons and ablation studies.

First, Table 2 provides comparisons with existing methods based on their published results or our reproduced evaluations. Since prior methods report results on different backbone configurations, baseline coverage naturally varies across model families. For Qwen3.5, we were not aware of directly comparable published streaming methods on the evaluated backbones at submission time, so we reported the base-model results as a reference.

Second, the requested comparisons are already addressed through controlled ablations on representative backbones. Appendix E.1, Tables 14 and 15 evaluate Qwen2.5-VL-7B and Qwen3-VL-8B across fixed-α\alpha settings, where α=0\alpha=0 and α=1\alpha=1 correspond exactly to Semantic-only and Recent-only, respectively. WRWS outperforms the best tested fixed weighting on both backbones.

ModelAllocationBak.RT.Fwd.B+RAll
Qwen2.5-VL-7Bα=0.00\alpha=0.00 (Semantic-only)47.1259.0839.1753.1048.46
α=0.25\alpha=0.2547.9764.4640.7256.2151.05
α=0.50\alpha=0.5050.2268.3439.9659.2852.84
α=0.75\alpha=0.7549.9773.9538.5861.9654.16
α=1.00\alpha=1.00 (Recent-only)50.4377.2039.5663.8255.73
Adaptive50.5977.1340.2363.8655.98
Qwen3-VL-8Bα=0.00\alpha=0.00 (Semantic-only)48.4662.8743.9555.6751.76
α=0.25\alpha=0.2548.8069.7245.9059.2654.81
α=0.50\alpha=0.5049.5673.3245.5761.4456.15
α=0.75\alpha=0.7549.9078.3946.4764.1458.25
α=1.00\alpha=1.00 (Recent-only)49.4980.7044.1165.0958.10
Adaptive50.0080.7445.0565.3758.59

Furthermore, within the limited rebuttal timeframe, we prioritized the reviewer's concern and conducted additional ablations on Qwen3.5-35B-A3B. As shown in the table below, WRWS achieves 64.8% overall accuracy, matching the best fixed setting of α=1.0\alpha=1.0 at the reported precision, without backbone-specific tuning.

AllocationBak.RT.Fwd.B+RAll
α=0.00\alpha=0.00 (Semantic-only)58.366.046.562.256.9
α=0.25\alpha=0.2560.171.647.165.859.6
α=0.50\alpha=0.5061.876.847.469.362.0
α=0.75\alpha=0.7563.080.347.671.663.6
α=1.00\alpha=1.00 (Recent-only)65.583.045.974.264.8
Adaptive65.182.846.473.964.8

These additional results further strengthen our cross-backbone evaluation and show that the same uncertainty-adaptive rule remains competitive with the best fixed policy across backbones without backbone-specific tuning.


Q4 — Requirements for AKS and FOCUS comparisons under streaming constraints

We appreciate the reviewer's suggestion, but would like to emphasize a fundamental distinction: causal frame availability alone does not ensure a fair efficiency comparison between offline keyframe selection and streaming video understanding.

To our knowledge, neither AKS [1] nor FOCUS [2] has been designed or evaluated for causal streaming video understanding. Both target offline video-query pairs, where keyframe selection operates on an available video conditioned on the query. In streaming settings, however, visual observations arrive before the query, so query-dependent selection cannot be completed in advance. Applying their original offline selection pipelines would therefore introduce query-time selection costs, which are particularly important for real-time responsiveness.

In contrast, WRWS is explicitly designed for incremental streaming processing. It maintains lightweight visual representations as observations arrive, enabling query-time evidence allocation from cached features before encoding only KK selected observations with the target Video-LLM.

Therefore, matching causal frame availability and the KK-frame budget alone is insufficient to establish equivalent streaming efficiency. A meaningful comparison must also account for the computational schedule, particularly streaming-side processing and query-time selection latency, alongside downstream accuracy.

[1] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive Keyframe Sampling for Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29118–29128, 2025.

[2] Zirui Zhu, Hailun Xu, Yang Luo, Yong Liu, Kanchan Sarkar, Zhenheng Yang, and Yang You. FOCUS: Efficient Keyframe Selection for Long Video Understanding. In The Fourteenth International Conference on Learning Representations, 2026.

I would like to thanks the auther to answer my quesiton, but seems the response donot fully resove my concern. Thanks the auther for add more VL baselines and comparasion, but for Q2 &Q3, we can see the gain is still limited among different VL models compared with recent-only method. While the changed-selection subset analysis helps explain where the gains arise, it does not establish a larger overall benefit. My concern is the improvement over Recent-only on the full evaluation set, which remains small. Regarding Q4, I understand that AKS targets offline video. However, its selection strategy could be adapted to causally available frames under the same frame budget, replacing your adaptive semantic–recency scoring and Top-K selection (Eqs. 7–8) with AKS’s relevance–coverage selection strategy to compare accuracy and selection latency.

Reviewer_7GAtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of allocating limited visual computing resource to visual encoding for LLM-based stream video understanding. The key idea is to develop a principled way to decide whether to retrieve/ find evidence from past semantics vs. recent streams; and that principled way is designed as an uncertainty gauge based on the distribution of event-query similarity. The proposed solution consists of a lightweight query-observation retrieval, uncertainty-aware temporal evidence allocation and selective target-VLM encoding. Experimental results based on multiple VLMs as backbones on a few benchmarks show the effectiveness of the proposed solution especially on efficiency.

Critical strengths
  1. The paper is presented clearly with well-crafted system architecture; the logic of the method and experiments is easy to follow.
  2. Experiment study with multiple VLM backbones and comprehensive evaluation/comparison with baselines.
Critical weaknesses

The main issue of the proposed method lies in its unrectified usage of uncertainty as a proper gage – I call it “recency hypothesis”- when there is no clear peak of past segments (uncertain), the method will choose to retrieve from the “recent” stream. However, such a hypothesis may not hold, despite some efforts to validate it and some empirical evidence based on selected benchmark tasks. While the presence of one or a few peaks of similarity scores can be considered as presence of contextual evidence that can be used to retrieve the related past streams, the absence of it (i.e. high uncertainty) needs not serve as evidence that such information is not from the past. Simply relying on recency for high-uncertainty cases is nothing but a last resort or a convenient guess. The positive evidence observed in the experiments (including the ablation study) could be attributed to the QA distribution in the specific benchmarks. Therefore, the proposed strategy may be only valid for the selected benchmarks. Without systematic justification/validation of the recency hypothesis, the methodological validity/significance is questionable. Related to this, an alternative simple strategy (as a benchmark) is to do opportunistic retrieval (i.e. randomly/uniformly sample streams within the budget) from the past + recent. The second weakness relates to the weak performance gain of the methodology compared to the benchmarks. Despite the extensive experiments, the actual gain from the proposed method is not substantial (sometimes mixed). Although there is substantial gain in efficiency, it is inadequate to justify the value of the method.

Questions for authors

How to justify the that high-uncertainty is sufficient condition for recency-based retrieval, while overcoming the potential biases in the benchmarks? I suggested an alternative opportunistic strategy (random or uniform) retrieval. Any justifications why it is not a good/valid benchmark? Otherwise, what is the comparative performance of this strategy vs. proposed method?

9 comments

Responses to Reviewer #7GAt (1/2)

W1 & Q1 — Justification for favoring recent evidence under high retrieval uncertainty

To better clarify the rationale behind our evidence-allocation design, we would like to emphasize a key distinction: recency is not a fallback when semantic retrieval fails, but an empirically established strong prior that serves as the starting point of WRWS. Our goal is to selectively incorporate historical evidence without unnecessarily compromising this strong prior. As demonstrated by SimpleStream [1] and further confirmed in our Table 1 (shown below), Recent-only outperforms Semantic-only by 7.27 and 6.34 points on Qwen2.5-VL-7B and Qwen3-VL-8B, respectively. However, Recent-only inherently misses evidence outside its retained window, while the per-query oracle reveals substantial complementarity between the two policies (+6.47/+7.65 points over Recent-only).

BackbonePolicyBackwardReal-TimeForwardAvg.
Qwen2.5-VL-7BSemantic-only47.1259.0839.1748.46
Recent-only50.4377.2039.5655.73
Oracle59.2181.7745.6362.20
Qwen3-VL-8BSemantic-only48.4662.8743.9551.76
Recent-only49.4980.7044.1158.10
Oracle60.5684.4552.2565.75

Thus, the role of uncertainty is not to establish whether relevant evidence is recent, but to assess when semantic relevance is sufficiently discriminative to justify shifting away from the strong recency prior. Specifically, WRWS computes the evidence score as Fi=(1−αt)s^i+αtriF_i=(1-\alpha_t)\hat{s}_i+\alpha_t r_i, where αt\alpha_t denotes retrieval uncertainty. This soft interpolation progressively strengthens the recency prior as uncertainty increases, without assuming that historical evidence is irrelevant.


W2 — Main concern about potential benchmark-distribution bias in uncertainty-guided allocation

We thank the reviewer for raising this important concern. Our empirical evidence extends beyond a single benchmark or task distribution, as demonstrated by both cross-benchmark evaluations and within-task uncertainty analysis.

First, as shown in Table 2 (shown below), WRWS achieves competitive performance on both StreamingBench and OVO-Bench across multiple Video-LLM backbones. For instance, with Qwen3-VL-8B, WRWS improves over SimpleStream from 80.59 to 81.11 on StreamingBench and from 57.55 to 58.59 on OVO-Bench.

Model#FramesStreamingBench Real-TimeOVO-Bench BackwardOVO-Bench Real-TimeOVO-Bench ForwardOVO-Bench Avg.
Qwen3-VL-8B1 fps80.5144.0065.0046.3051.77
SimpleStream1 fps80.5948.8481.2642.5557.55
WRWS (Ours)1 fps81.1150.0080.7445.0558.59

Second, the uncertainty-dependent advantage of Recent-only over Semantic-only persists after accounting for task-level distributional differences (Appendix E.2, Table 16, shown below). Specifically, we partition queries into uncertainty quintiles within each of the 12 OVO-Bench tasks and then aggregate the corresponding quintiles across tasks. The advantage of Recent-only over Semantic-only increases from only 0.82 points in the lowest-uncertainty quintile to 9.59 and 7.82 points in the two highest quintiles, respectively, with both differences having 95% confidence intervals excluding zero.

Uncertainty# QueriesSemantic-onlyRecent-onlyΔS−R\Delta_{\mathrm{S-R}}95% CI
Q1 (lowest)61262.0962.91-0.82[-3.93, 2.45]
Q260952.7155.83-3.12[-6.57, 0.16]
Q360849.1852.63-3.45[-6.91, 0.00]
Q460543.1452.73-9.59[-13.39, -5.79]
Q5 (highest)60141.6049.42-7.82[-11.48, -4.16]

Together, these results provide complementary evidence for the effectiveness of WRWS across benchmarks and further support the empirical motivation for uncertainty-guided allocation, with the observed uncertainty-dependent pattern persisting after accounting for task-level distributional differences.


Responses to Reviewer #7GAt (2/2)

W3 & Q2 — Comparison with random and uniform sampling baselines

We agree that random and uniform sampling are meaningful baselines and appreciate the reviewer's suggestion. Accordingly, we conduct additional experiments with random and uniform sampling over all causally available observations, covering both historical and recent frames, on Qwen2.5-VL-7B and Qwen3-VL-8B. For random sampling, we report the mean and standard deviation across three independent random seeds.

As shown in the table below, WRWS consistently outperforms both sampling strategies across the two backbones, demonstrating that simply sampling observations from the video history is less effective than our uncertainty-adaptive evidence allocation. These results further support the importance of preserving a strong recency prior while selectively incorporating semantically relevant historical evidence.

BackboneMethodBack.RTFwd.B+RAll
Qwen2.5-VL-7BRandom44.68±0.4044.68 \pm 0.4050.98±1.9650.98 \pm 1.9633.47±0.4833.47 \pm 0.4847.83±1.1547.83 \pm 1.1543.04±0.9243.04 \pm 0.92
Uniform46.5166.4935.2456.5049.41
SimpleStream50.4377.2039.5663.8255.73
WRWS (Ours)50.5977.1340.2363.8655.98
Qwen3-VL-8BRandom47.27±0.5147.27 \pm 0.5153.25±1.3453.25 \pm 1.3438.46±0.7738.46 \pm 0.7750.26±0.7750.26 \pm 0.7746.33±0.5746.33 \pm 0.57
Uniform48.9370.1242.0659.5253.70
SimpleStream48.8481.2642.5565.0557.55
WRWS (Ours)50.0080.7445.0565.3758.59

W3 & Q3 — Practical value and accuracy–efficiency trade-off of WRWS

We emphasize that computational efficiency is a fundamental requirement of streaming video understanding, rather than merely a secondary optimization objective. Unlike offline video understanding, streaming systems must continuously accommodate incoming observations while answering queries under constrained computational resources; otherwise, the accumulated video history could simply be processed as an offline clip at query time. WRWS directly addresses this challenge by incrementally maintaining lightweight retrieval representations and selecting a bounded set of observations for expensive target-VLM encoding, rather than encoding every incoming observation.

Importantly, WRWS achieves a favorable accuracy–efficiency trade-off. As shown in the table below, WRWS reduces VTAT by 52.1% (a 2.09× speedup) relative to the evaluated encode-first SimpleStream baseline, while improving accuracy on both StreamingBench and OVO-Bench. Figure 4 of the original paper further demonstrates that the relative computational savings increase with stream length. Moreover, uncertainty-adaptive allocation enables selective access to relevant historical evidence under the same target-model visual budget, without backbone-specific weight tuning. Together, these results highlight the value of WRWS in substantially reducing cumulative visual-processing cost while preserving competitive downstream performance and effective historical evidence retrieval.

MethodStreamingBench Accuracy (%) ↑OVO-Bench Accuracy (%) ↑VTAT (s) ↓Reduction ↑Speedup ↑
SimpleStream78.4755.7312.18—1.00×
WRWS (Ours)79.4755.985.8452.1%2.09×

[1] Yujiao Shen, Shulin Tian, Jingkang Yang, and Ziwei Liu. A Simple Baseline for Streaming Video Understanding. arXiv preprint arXiv:2604.02317, 2026.

I'd like to thank the authors on the answers to address my questions. Regarding W1/Q1, the authors rightly confirmed my previous assumption, i.e. the usage of recency when semantic-retrieval fails is indeed based a strong prior of recency preference which is inherent in the statistics of benchmarks (rather than a common empirical of video retrieval in general). This is exactly where my concern lies in - a low uncertainty score tells us that the model cannot detect/find relevant information to answer a question, but it is not a sufficient condition to tell that such information does not exist in the semantics. As such, resorting to recency is dependent of the statistical prior and will has limited value when the distribution changes in, say, a different benchmark. As such, the methodological value of the proposed solution is not adequately justified. Regarding W2/Q2, I understand there is improvement compared to baselines. However, the gap is not substantial, e.g. compared to Simplestream, the improvement is 2.5% max, and mostly around 1%.

Comments on the paper

1 comment

General Responses

We sincerely thank all reviewers for their constructive feedback and for recognizing the practical importance, training-free and model-external design, broad applicability, comprehensive evaluation, and substantial efficiency gains of WRWS. We address their concerns through the following clarifications and additional results. Key updates include:

  1. Importance of Efficiency: Streaming video understanding requires continuously processing incoming observations while responding promptly to user queries, making computational efficiency a fundamental requirement. (Reviewers #7GAt, #7nbS)

  2. Accuracy–Efficiency Trade-off: WRWS achieves competitive or improved accuracy while reducing VTAT by 52.1% (2.09× speedup) over the evaluated encode-first SimpleStream baseline. (Reviewers #7GAt, #Kxnf, #7nbS)

  3. Methodological Motivation and Contributions: We further justify uncertainty-adaptive allocation, clarify recency as a strong prior rather than a fallback, and distinguish our retrieve-first design from existing approaches. (Reviewers #7GAt, #Kxnf, #7nbS)

  4. Additional Baselines: We add random and uniform sampling baselines on Qwen2.5-VL-7B and Qwen3-VL-8B, further validating our adaptive allocation strategy. (Reviewer #7GAt)

  5. Further Adaptive-Allocation Analysis: We provide within-task uncertainty analysis and additional results on queries where WRWS differs from Recent-only, revealing settings where adaptation yields meaningful gains. (Reviewers #7GAt, #Kxnf)

  6. Expanded Backbone Evaluation: We extend fixed-weight ablations to Qwen3.5-35B-A3B to further examine the behavior of the adaptive strategy across backbones. (Reviewers #Kxnf, #7nbS)

  7. Efficiency Evaluation: We clarify the scope of VTAT, discuss buffered Recent-only, and distinguish cumulative streaming-processing cost from query-time latency. (Reviewers #7GAt, #7nbS)

We hope these clarifications and additional results better demonstrate the methodological and practical significance of WRWS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.