- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors address the challenge of perceiving fast-paced, high-dynamic events in continuous video streams under a bounded context budget. The authors introduce FastBench, a streaming video question-answering (QA) benchmark consisting of 306 QA pairs across 8 domains, 6 capability dimensions, and 3 temporal scopes (Forward, Instant, and Backward).
Critical strengths
- The paper identifies a genuine, practical bottleneck in streaming VLMs, the tension between fine-grained temporal resolution (high FPS) and long-term historical retention under a fixed visual token budget. The proposed curation pipeline is a standout contribution: combining a 2-FPS answerability filter with explicit physical trajectory verification (via SAM3 and CoTracker3) and multi-round human inspection effectively eliminates static visual shortcuts and mitigates VLM hallucinations during annotation of fast-paced motion.
- The ablation studies clearly expose the fundamental trade-off in bounded-context streaming perception. Showing that Qwen3-VL-8B scales monotonically from 2 FPS (32.9%) to 24 FPS (44.6%) before dropping at 30 FPS (42.1%)—primarily driven by a sharp degradation on Backward queries (46.2% to 37.5%) as historical context shrinks to 4 seconds—provides concrete evidence for why adaptive temporal sampling is necessary in streaming systems.
Critical weaknesses
- While the filtering pipeline is rigorous (reducing 16,702 candidates to 306 samples), the final dataset of 306 QA pairs is small when partitioned across 6 capabilities, 8 domains, and 3 temporal scopes. In particular, the "Streaming & Online Detection" category accounts for only 2.61% of the dataset (merely 8 QA pairs), and backward/instant scopes contain only 65 and 73 samples, respectively. This makes fine-grained sub-category claims, such as highlighting a "+37.5% surge in the Streaming & Online category" in Section 4.2, which corresponds to roughly 3 samples, statistically fragile.
- Under the exact same 120-frame context budget, ProactiveFrame underperforms simple uniform higher-FPS sampling. Specifically, in Table 2, Full 30 FPS outperforms ProactiveFrame on both Qwen3-VL-8B (42.1% vs. 38.3%) and Gemini-3.5-Flash (55.3% vs. 52.2%); in Table 3, even uniform 4 FPS (38.6%) and 8 FPS (40.5%) outperform ProactiveFrame (38.3%) on Qwen3-VL-8B while preserving 30s and 15s of history, respectively. Moreover, because ProactiveFrame only increases the frame rate of incoming chunks after outputting
<Focus Start>on a 1-second low-FPS chunk, it suffers from an inherent causal lag: for transient events lasting $\le$1s (e.g., the capacitor spark in Figure 1) or Backward/Instant queries where the critical action occurs before the model triggers focus mode, high-FPS sampling arrives too late.
Questions for authors
- Given that uniform 4–30 FPS sampling consistently outperforms ProactiveFrame under the same 120-frame budget (Tables 2 and 3), how frequently do models actually trigger
<Focus Start>during evaluation, and what is the precision/recall of their triggered intervals against the human-annotated evidence intervals ? Additionally, how can a forward-looking trigger help on Backward queries (where ProactiveFrame shows +4.6% and +5.2% gains in Table 1) if the query arrives after the evidence interval has already passed? - Could you clarify the exact evaluation details in Section 4.1? Specifically: (a) Which model is used as the LLM-as-a-Judge, and what is the scoring rubric (e.g., 0–5 or 0–10 scale, which would explain non-multiples of 12.5% like 2.5% and 42.5% on the 8-sample "Streaming & Online" subset)? (b) What are the exact input FPS and context window settings for all baseline models in Table 1 (especially native streaming models like JoyAI-VL-Interaction and MOSS-VL-Realtime)?
- Since Gemini-3.1-Pro performs the 2-FPS answerability filtering, questions that Gemini fails to answer at 2 FPS are retained, while questions it answers correctly at 2 FPS are discarded. Did the authors evaluate whether this introduces a selection bias that favors or penalizes the Gemini family relative to other model families?
3 comments
W2 / Q1: Autonomous focus, causal delay, and Backward questions
We agree with the reviewer's comparison: autonomous ProactiveFrame does not outperform the cited higher-FPS baselines in overall score. Qwen3-VL-8B achieves 38.3, compared with 38.6 at 4 FPS, 40.5 at 8 FPS, 44.6 at 24 FPS, and 42.1 at 30 FPS; Gemini-3.5-Flash achieves 52.2 versus 55.3 at 30 FPS. The demonstrated gains are relative to the respective 2-FPS baselines. We will state this scope explicitly and characterize ProactiveFrame as a training-free baseline for studying observation control.
How often is focus triggered, and how well does it overlap the evidence? We audited the saved focus logs. Among clips eligible for the interval analysis, Qwen emits at least one Focus_Start in 250/261 clips (95.8%), and Gemini in 276/278 clips (99.3%). Their mean numbers of emitted start commands are 4.61 and 5.65 per clip, respectively.
For predicted focus intervals P, annotated focus intervals G, and intersection duration I, we measure precision as I/|P|, coverage/recall as I/|G|, IoU as I/|P ∪ G|, and F1 as 2I/(|P|+|G|). Metrics are computed per clip and then macro-averaged; all values below are percentages.
| Model | Eligible clips | IoU | Coverage / recall | Precision | F1 |
|---|---|---|---|---|---|
| Qwen3-VL-8B | 261 | 19.38 | 35.53 | 28.39 | 26.36 |
| Gemini-3.5-Flash | 278 | 29.93 | 61.05 | 35.55 | 40.63 |
Both predicted and GT intervals are clipped to a 10-second window starting at the GT start for Forward, or ending at the GT end for Backward/Instant.
Why can Backward questions benefit? The focus policy operates before a question arrives. The prompt explicitly asks the model to monitor the stream even without an active question and request denser observations when an unfolding event appears likely to require fine temporal detail. For a Backward question, any useful dense evidence must therefore have been captured proactively before that question and remain available when it is answered. The +4.6 and +5.2 point Backward gains over 2 FPS are compatible with this mechanism.
This is harder than the Forward setting, where a known question can direct attention to a target event. We provide the following descriptive comparison under the same analysis procedure:
| Temporal-scope group | Eligible clips | IoU | Coverage / recall | Precision | F1 |
|---|---|---|---|---|---|
| Forward | 156 | 34.69 | 72.55 | 38.29 | 45.99 |
| Backward | 64 | 25.01 | 42.40 | 35.43 | 35.96 |
| Instant | 61 | 23.31 | 51.15 | 29.72 | 32.35 |
All four Backward metrics are below Forward, consistent with the difficulty of choosing relevant observations without knowing the future query.
The causal-delay limitation is real. A command emitted after processing a one-second chunk changes sampling only for subsequent chunks. If a brief event ends before focus starts, the policy cannot retrospectively obtain its high-FPS evidence. This applies to an unpredictable one-second spark and to evidence already missed before Backward/Instant queries. Benefits require anticipatory cues, an event that continues into later chunks, or focus already being active. We will explicitly discuss this failure mode.
Finally, annotation-guided focus raises Gemini's overall score to 57.5, compared with 52.2 for autonomous focus and 55.3 for Full 30 FPS. This supports the potential of allocating fine temporal observations to relevant intervals while retaining sparse history. It is an annotation-assisted reference, not a deployable policy or a proven optimal allocation. The benefit is also backbone-dependent: Qwen's oracle score of 40.7 remains below Full 30 FPS (42.1). We will distinguish the potential of selective observation from the limitations of the present autonomous controller.
Q2: Judge, scoring rubric, FPS, and context settings
(a) Open-ended scoring. We follow PhoStream's open-ended evaluation framework. The specified evaluation launcher selects Qwen3-235B-A22B-2507 as the judge. It receives the question, model answer, and reference answer, and is instructed to return an integer score from 0 to 5:
| Score | Rubric |
|---|---|
| 5 | Fully accurate and complete |
| 4 | Correct and sufficient |
| 3 | Partially relevant |
| 2 | Tangential/speculative |
| 1 | Factually incorrect |
| 0 | No attempt/off-topic |
The rubric accepts coherent, factually plausible answers without requiring every detail or the exact wording of the reference. Each eligible answer's score is multiplied by 20 and then averaged across QAs. Consequently, these percentages are normalized mean scores, not binary accuracy. On an eight-QA subset, one judge-point change on one answer changes the mean by 20/8 = 2.5 percentage points, directly explaining values such as 2.5 and 42.5.
(b) Sampling and context. The common baseline launch configuration specifies a maximum context budget of 248,832,000 pixels. In the shared video-context path, this retains approximately 60 seconds at 2 FPS. The paper's pixel-budget specification, 248,832,000 = 120 × 1920 × 1080, is a 120-frame-at-1080p equivalent. We will distinguish frame caps, pixel budgets, and native history policies, and qualify comparisons accordingly rather than describe all Table 1 rows as strictly context-matched.
Q3: Gemini-based filtering and possible selection bias
The 2-FPS answerability filter serves FastBench's central requirement: questions should depend on high-dynamic visual evidence that sparse temporal sampling can miss. Questions that Gemini-3.1-Pro answers correctly at 2 FPS are excluded because they can already be resolved under the low-FPS setting and therefore do not meet this selection criterion.
The potential model-dependent bias arises among the retained candidates: a question that Gemini cannot answer at 2 FPS might still be answerable by another model from the same sparse observations. Our final human quality check specifically re-examines whether each question requires fine-grained temporal evidence, including whether the answer can instead be obtained from sparse frames or static visual cues. Candidates that do not satisfy this high-dynamic requirement are excluded at this stage. This human review provides an additional check on the relevance of retained questions to the benchmark's objective. We will make these human quality-control criteria explicit in the revised construction description, particularly the requirement to verify dependence on high-dynamic evidence in addition to answer correctness and visual clarity.
We thank the reviewer again for the thoughtful feedback and look forward to any further comments on our clarifications and additional results.
We thank the reviewer for recognizing the practical motivation, the annotation pipeline, and the temporal-resolution/history trade-off. The concerns about small subsets, autonomous focus selection, and evaluation transparency are well taken. We address them below and will revise the corresponding claims and protocol descriptions.
W1: Dataset size and the interpretation of small subsets
FastBench's final scale and its capability and temporal-scope distributions emerge from broad candidate collection followed by our quality and high-dynamic-relevance filters. The pipeline selects 306 carefully verified QAs across eight domains from 16,702 candidates, prioritizing visual clarity and dependence on high-dynamic information. This selective curation supports the benchmark's purpose: diagnosing whether streaming models can capture brief temporal evidence and retain it for subsequent questions.
The Streaming & Online Detection subset contains 8 QAs (2.61%), while Backward and Instant contain 65 and 73 QAs, respectively. These category-level results provide descriptive diagnostics of model behavior on distinct streaming tasks. To express the observed improvement precisely, we will revise “+37.5% surge” to state that the mean normalized score increases from 2.5 to 40.0, a gain of 37.5 percentage points on the eight-QA Streaming & Online Detection subset. As detailed in Q2, each answer receives a graded open-ended score, the reported gain therefore reflects changes in answer quality across the subset. We will report category counts alongside these scores to make the basis of each comparison explicit.