acceptodds
Withdrawn from ICLR 2027

Q-Span: Query-Adaptive Look-Back Windows for Streaming Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

What a streaming video assistant can answer is bounded by how much history it keeps, but decided by which slice of that history the model is actually shown, and that slice cannot be fixed. Measuring accuracy against the look-back window on three streaming benchmarks, we find the window a question needs varies at least in length, and that populations disagree about it as sharply *within* a benchmark as between them: on one level of OVO-S-Bench a s window beats a s one by points, and on another the longer window wins by . No single window serves a stream of mixed questions. We propose Q-Span, which reads the question and nothing else to choose the span. Which of three windows a question calls for, a close look at the present, a recent span, or a thin pass over the whole prefix, is settled by its wording, so one text call picks the window before a frame is decoded. How far back the evidence sits is not settled by the wording: the same sentence can sit three seconds from its answer in one recording and three minutes in another. That residue is read off the frames, in one further call on frames the answer will use anyway. The method is training-free and its rules name no benchmark's templates, though written for the kinds of question streaming benchmarks ask: planner, probe and answering model are one frozen backbone under different prompts. Across three backbones and three benchmarks it holds both ends at once, which no fixed window does, and the gain follows the mixture: three points where a benchmark spreads its questions, and a tie where they do not.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, video question answering, look-back window, training-free, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_VUJVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select the look-back window for streaming video question answering. Through experiments on three benchmarks, the authors show that different question types favor substantially different temporal spans, making a single fixed window unsuitable for mixed queries. They propose Q-SPAN, a method that requires no additional training and uses the same frozen vision-language model for planning, evidence checking, and answering. A planner receives only the question text and selects among three sampling policies: the last 4 seconds, the last 32 seconds, or up to 128 frames sampled uniformly across the observed video prefix. For questions assigned to the shortest window, a visual probe checks whether the sampled frames contain sufficient evidence and expands the window to 32 seconds if needed. Experiments across three backbones show improved accuracy over fixed sampling policies, with the clearest gains on benchmarks containing a mixture of current-view and historical questions. The advantage over the best fixed window becomes small when one temporal regime dominates

Critical strengths

The paper systematically examines the relationship between look-back span and accuracy across multiple benchmarks and question subsets. The results show that aggregate performance can conceal opposing temporal requirements across question types.

Critical weaknesses
  1. A local probe may mistake relevant evidence for sufficient evidence, particularly for questions such as “Is this the first time?” that require checking earlier occurrences. The paper does not provide an ablation by question type showing where the probe helps, provides no gain, or harms accuracy, leaving its reliability across these cases unclear.

  2. The improvement over the best fixed window is clearest on OVO-S-Bench, while the corresponding gains on OVO-Bench and StreamingBench’s real-time subset are small or negligible in several settings.

  3. Missing random frame-sampling baselines. The paper compares against shuffled window assignments, but does not evaluate randomly sampling a matched number of frames from the full prefix or a recent temporal window. These inexpensive baselines would help establish whether the proposed routing offers an advantage over simpler sampling strategies.

Questions for authors

See above

Reviewer_KRkVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the optimal look-back window for streaming video QA varies by up to 64× across question populations, with opposing trends even within one benchmark. It proposes Q-SPAN, a training-free policy in which the frozen backbone reads only the question to pick one of three windows (4 recent frames, 32 recent frames, or 128 frames over the whole prefix); if it picks the shortest, a one-token probe over those frames can widen it by one step. Across 7B–27B backbones, it beats prior training-free systems, gains 3.09 over the best fixed window on OVO-S-Bench, and ties elsewhere.

Critical strengths
  1. A clear, well-controlled empirical observation. Figure 1 and Tables 11, 16, 20, 21, and 23 show that the best window is population-specific. Table 23 holds the frame count fixed and varies only the span, which rules out frame count as the explanation.
  2. Honest, thorough ablations that locate the gain. The key results:
    • the label shuffle scores 48.13 against the planner's 52.33 at the same window mix and frame budget;
    • fixed windows tuned on held-out data still lose (Table 15);
    • the bootstraps (Tables 13–14) and the question-mix sweep (Fig. 3) show where the gain appears and where it vanishes;
    • Sec. 6 states the failure cases plainly.
  3. Cheap and practical. One backbone and one prompt work from 7B to 27B, and the planner and probe add 55–78 ms. Q-SPAN is more accurate than the best fixed window and also faster than it (TTFT 918 vs 2625 ms on OVO-S-Bench; 540 vs 625 ms on StreamingBench real-time). The planner's own traffic predicts whether routing will pay (Fig. 4, r = 0.96), without labels or frames.
Critical weaknesses
  1. Gains over the best fixed window are narrow.
    • Routing ties the strongest single window in 6 of 9 settings (Table 13), and on OVO-S-Bench 7B the confidence interval is [−0.26, +2.64].
    • At 8B, the entire +3.09 on OVO-S-Bench comes from L1 (+13.84; L2 −1.48; L3 and L4 tie). In effect, the gain comes from keeping current-view questions on a recent window, which is close to SimpleStream's finding.
    • Table 1's large margins are mostly against a weak [Base] protocol and against published systems with different budgets. On OVO-Bench real-time, a fixed 4-frame window already scores 82.3, against Q-SPAN's 82.4.
  2. The hand-written prompt is never tested on held-out data.
    • The paper does not say how its 5 rules and 11 examples were developed, and every evaluated benchmark also serves as the test set.
    • Rule R5 closely echoes OVO-S-Bench's own question templates (Table 27).
    • B.17's robustness checks help, but the authors concede they cannot rule out a different result on a new benchmark.
    • The prompt needs to be evaluated on an unseen streaming benchmark, e.g., OVBench or OmniMMI.
  3. Several numbers do not reconcile.
    • On OVO-S-Bench, [Base] is described both as "1 fps, up to 256 frames" and as the 128-frame uniform protocol. At 8B, it matches the widest window (49.9 vs 49.85). However, at 7B and 27B it does not: 44.2 and 54.0, against the 45.62 and 55.13 implied by Table 13. So the headline +16.3 / +6.8 in Sec. 5.2 is measured against a different baseline from the deployed window.
    • In Table 15, it gives +0.36 for StreamingBench real-time 7B, while Tables 13–14 give +0.08. This also shifts Table 15's mean.
  4. References are incomplete.
    • VST, StreamChat, LongVU, LLaVA-Video, Video-LLaMA2, and InternVL-3.5 have no bibliography entry.
    • StreamForest's author list is wrong.
Questions for authors
  1. How was the planner prompt developed: which benchmarks and how many iterations were inspected while writing the rules and examples? Can you report results on a streaming benchmark that was not consulted?
  2. What exact protocol produced [Base] on OVO-S-Bench at 7B and 27B (44.2 / 54.0)? What is Q-SPAN's gain over the deployed 128-frame uniform window there? Which of the Table 13 and Table 15 values for StreamingBench 7B is correct?
  3. On OVO-Bench, what are Q-SPAN's HLD and Forward Active Responding scores? How do Tables 13–14 change if the paired comparison covers real-time and backward tracing together, rather than real-time alone?
Reviewer_fWfQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-Span, a training-free framework for query-adaptive look-back window selection in streaming video understanding. A text-only planner selects one of three predefined windows based on the question, while a visual sufficiency probe optionally expands the shortest window. The framework uses the same frozen MLLM for planning, probing, and answering. Experiments across three streaming benchmarks and multiple backbones show that adaptive window selection can improve accuracy, particularly when questions require different amounts of historical evidence.

Critical strengths
  1. The paper identifies an important issue in streaming video understanding: different questions require substantially different historical observation ranges. The proposed text-first routing mechanism is simple, training-free, and does not require additional model weights or continuous memory construction.

  2. The paper evaluates multiple backbones and benchmarks and provides extensive ablations on fixed windows, routing strategies, visual probing, prompt design, and computational cost. The paired bootstrap analysis and the explicit discussion of settings where adaptive routing provides little benefit are particularly valuable.

Critical weaknesses
  1. The proposed framework primarily relies on prompting an existing MLLM to select one of three predefined windows, followed by a binary visual sufficiency check. While effective in certain settings, the methodological contribution is relatively incremental. Moreover, closely related work such as StreamScout (2026) has already investigated query-adaptive evidence acquisition through progressively richer visual views and sufficiency-based escalation in streaming video understanding. The key distinction of Q-Span appears to be text-only planning before visual inspection and single-pass answering, but its advantages over these existing approaches are not sufficiently established.

  2. According to Table 13, Q-Span significantly outperforms the strongest fixed-window baseline in only two of the nine benchmark-backbone settings. Most improvements on OVO-Bench and StreamingBench are statistically indistinguishable from zero. Furthermore, Table 6 reports a TTFT of 918 ms for Q-Span on OVO-S-Bench, compared with 301 ms for OASIS and 21 ms for the fixed short window. Although the method improves accuracy, its additional query-time cost is substantial, and the claimed efficiency benefits depend strongly on the comparison baseline and workload distribution.

  3. The approach assumes that the full video prefix remains accessible for query-time sampling, potentially requiring storage that grows with stream duration. The reported latency excludes video decoding and does not fully characterize this storage-access trade-off. In addition, the paired bootstrap analysis resamples questions rather than entire videos, despite potential dependencies among questions from the same video. The paper also needs to clarify the consistency of its evaluation protocol when comparing results from the 1,722 scored queries of the public OVO-S-Bench release with published results based on the benchmark's 1,680-item main evaluation set.

Questions for authors

Q1. StreamScout (2026) also investigates query-adaptive evidence acquisition for streaming video understanding using multiple visual views and sufficiency-based escalation. What are the fundamental differences between Q-Span and this approach, beyond using a text-only planner before visual inspection? Could the authors provide a more direct comparison or discussion of their respective accuracy-efficiency trade-offs?

Q2. Table 6 shows that Q-Span achieves 52.94% accuracy with 918 ms TTFT on OVO-S-Bench, compared with 46.8% and 301 ms for OASIS, and 47.74% and 21 ms for the fixed short window. How should these substantial latency differences be interpreted in a real-time streaming setting? Could the authors provide a more complete end-to-end evaluation including video decoding, frame retrieval, and memory usage?

Q3. The planner predicts a window category from the question alone, while the probe can only expand the short window to the medium one. However, Appendix Table 27 shows that identical question wording can correspond to evidence located between 9 and 706 seconds in the past. How does Q-Span handle questions requiring distant, scattered, or multiple historical events when the wording does not reveal their location? How robust is the approach to multi-turn follow-up questions?

Q4. Table 13 reports paired bootstrap confidence intervals based on resampling questions. Since multiple questions may originate from the same video and share correlated visual evidence, could the authors verify the conclusions using video-level clustered bootstrap resampling? This would help establish whether the reported improvements remain statistically reliable.

Q5. The paper evaluates 1,722 scored queries from OVO-S-Bench, while the benchmark's published main evaluation reports 1,680 items. Could the authors clarify which baseline results were reproduced on the same 1,722-query release and which were taken from previously published results? Additionally, Table 15 reports a hindsight gain of +0.36 points for StreamingBench RT with the 7B backbone, whereas Tables 13 and 14 report +0.08 points for the same setting. Could the authors clarify this inconsistency?

Reviewer_4pM5OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the choice of look-back window in streaming video question answering. They propose Q-SPAN: a single text-only call (planner) reads the question wording alone and picks one of three windows (4s / 32s / 128 frames over the full prefix); a further visual call (probe) runs only on the short path, checking whether the fetched frames contain the needed evidence and escalating one step if not. The key insight: wording determines the kind of evidence a question refers to (categorical), but not how far back it sits (metric) . The method is training-free, with planner, probe and answering model being one frozen backbone under different prompts.

Critical strengths
  1. The paper provides a comprehensive set of ablations, with targeted controls constructed for each key design choice: by shuffling the planner's labels across questions while keeping the frame count and window mixture fixed (accuracy drops from 52.33 to 48.13), the authors show that the gain comes from the per-question pairing rather than from the frame budget or the window ratio; and by replacing the planner with a regex baseline and by asking the planner to predict a duration in seconds, they rule out the alternative explanations of keyword matching and reading the evidence distance from the wording, respectively, giving direct experimental support to the paper's central claim.

  2. The paper offers a thorough and honest analysis of the method's boundaries: for example, it explicitly identifies that the medium rung reaches only 32 seconds, so evidence lying between it and the full prefix falls into a gap between the two windows, and it quantifies the cost of this gap ; it further measures how the gain varies with the question mixture, and provides a pre-deployment diagnostic that requires only a single text-only call . This willingness to report the method's failure conditions up front, and to give readers the means to judge its applicability, is commendable.

Critical weaknesses
  1. The comparisons in Table 1 are not entirely fair in terms of frame budget: Q-SPAN spends 20–60 frames per question on average, whereas OASIS uses only 16 and the fixed short window uses 4, so readers cannot tell from the main table alone whether the reported advantage comes from the routing mechanism itself or simply from spending roughly three times as many frames.

  2. The planner's rules were summarized and written based on the question styles of the three benchmarks evaluated in this paper, so the generality of the proposed method remains unverified: while the authors take care to show that the prompt names no benchmark-specific templates and test its robustness to option order, demonstration removal, and re-wired window widths, all of these checks are still performed within the same three benchmarks, and no evidence from any fourth benchmark or real-world deployment is provided to confirm that the three-regime taxonomy itself transfers beyond the distribution it was designed around.

Questions for authors
  1. The main comparison in Table 1 evaluates each system at its own frame budget. Could you report a comparison across all methods under matched configurations, especially at the same frame budget, so that readers can directly assess whether the advantage comes from the routing mechanism itself rather than from spending more frames?

  2. For questions that involve two points in time — for example, "The second time I passed through the kitchen, were the things that had been on the table during my first pass still there?" — how does the current planner classify them, and how should they ideally be classified?

  3. The current method has several boundary-level gaps: questions whose evidence lies between 32 seconds and the full prefix are easily missed; and for long videos, the fixed 128-frame budget of the widest window becomes too sparse to answer questions precisely (and doubling the budget eventually exceeds the context limit). Do you see further solutions to these problems in future work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.