acceptodds
Withdrawn from ICLR 2027

FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming Video Large Language Models (VLMs) have emerged as a promising paradigm for real-time, continuous video understanding. However, existing streaming benchmarks predominantly target low-dynamic scenarios, masking a critical challenge: under a bounded context budget, a model must trade off long temporal history, spatial resolution, and temporal granularity, and the sparse sampling rates (1–2 FPS) adopted by current systems inevitably miss fast-paced events. To bridge this gap, we introduce FastBench, a benchmark designed to evaluate streaming VLMs on high-dynamic real-world video streams. FastBench is built with a trajectory-grounded pipeline: a VLM proposes candidate QA pairs from native high-frame-rate clips, a low-FPS filter discards pseudo-dynamic questions that remain answerable at 2 FPS, and a Vision Expert Verification stage grounds answers on object trajectories extracted by SAM3 and CoTracker3 to suppress hallucinations, followed by three rounds of human inspection. The resulting 306 QA pairs span eight domains (e.g., sports, gaming, wildlife, and transportation), six capability dimensions, and forward, instant, and backward temporal scopes, each with human-annotated evidence intervals. Alongside the benchmark, we present ProactiveFrame, a training-free baseline that lets a model raise or restore the frame rate of incoming chunks through text tokens, supported by a dual-tier sliding window that buffers recent high-FPS chunks and gracefully degrades them into sparse history. Experiments show that high-dynamic perception remains far from solved: the strongest model, Gemini-3.5-Flash, reaches only 50.7%. Denser sampling helps substantially (Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS) but saturates as history is compressed. ProactiveFrame improves over sparse uniform sampling (+5.4% and +1.5%), yet stays well below oracle-guided focusing, revealing that current VLMs struggle to decide, from the visual stream alone, when finer temporal perception is needed. FastBench establishes a rigorous testbed and offers actionable insights for high-dynamic streaming video understanding.

Primary area:
datasets and benchmarks
Keywords:
Streaming VLM, Proactive Perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MactOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address the challenge of perceiving fast-paced, high-dynamic events in continuous video streams under a bounded context budget. The authors introduce FastBench, a streaming video question-answering (QA) benchmark consisting of 306 QA pairs across 8 domains, 6 capability dimensions, and 3 temporal scopes (Forward, Instant, and Backward).

Critical strengths
  • The paper identifies a genuine, practical bottleneck in streaming VLMs, the tension between fine-grained temporal resolution (high FPS) and long-term historical retention under a fixed visual token budget. The proposed curation pipeline is a standout contribution: combining a 2-FPS answerability filter with explicit physical trajectory verification (via SAM3 and CoTracker3) and multi-round human inspection effectively eliminates static visual shortcuts and mitigates VLM hallucinations during annotation of fast-paced motion.
  • The ablation studies clearly expose the fundamental trade-off in bounded-context streaming perception. Showing that Qwen3-VL-8B scales monotonically from 2 FPS (32.9%) to 24 FPS (44.6%) before dropping at 30 FPS (42.1%)—primarily driven by a sharp degradation on Backward queries (46.2% to 37.5%) as historical context shrinks to 4 seconds—provides concrete evidence for why adaptive temporal sampling is necessary in streaming systems.
Critical weaknesses
  • While the filtering pipeline is rigorous (reducing 16,702 candidates to 306 samples), the final dataset of 306 QA pairs is small when partitioned across 6 capabilities, 8 domains, and 3 temporal scopes. In particular, the "Streaming & Online Detection" category accounts for only 2.61% of the dataset (merely 8 QA pairs), and backward/instant scopes contain only 65 and 73 samples, respectively. This makes fine-grained sub-category claims, such as highlighting a "+37.5% surge in the Streaming & Online category" in Section 4.2, which corresponds to roughly 3 samples, statistically fragile.
  • Under the exact same 120-frame context budget, ProactiveFrame underperforms simple uniform higher-FPS sampling. Specifically, in Table 2, Full 30 FPS outperforms ProactiveFrame on both Qwen3-VL-8B (42.1% vs. 38.3%) and Gemini-3.5-Flash (55.3% vs. 52.2%); in Table 3, even uniform 4 FPS (38.6%) and 8 FPS (40.5%) outperform ProactiveFrame (38.3%) on Qwen3-VL-8B while preserving 30s and 15s of history, respectively. Moreover, because ProactiveFrame only increases the frame rate of incoming chunks after outputting <Focus Start> on a 1-second low-FPS chunk, it suffers from an inherent causal lag: for transient events lasting $\le$1s (e.g., the capacitor spark in Figure 1) or Backward/Instant queries where the critical action occurs before the model triggers focus mode, high-FPS sampling arrives too late.
Questions for authors
  1. Given that uniform 4–30 FPS sampling consistently outperforms ProactiveFrame under the same 120-frame budget (Tables 2 and 3), how frequently do models actually trigger <Focus Start> during evaluation, and what is the precision/recall of their triggered intervals against the human-annotated evidence intervals [tstart,tend][t_{\text{start}}, t_{\text{end}}]? Additionally, how can a forward-looking trigger help on Backward queries (where ProactiveFrame shows +4.6% and +5.2% gains in Table 1) if the query tqt_q arrives after the evidence interval tendt_{\text{end}} has already passed?
  2. Could you clarify the exact evaluation details in Section 4.1? Specifically: (a) Which model is used as the LLM-as-a-Judge, and what is the scoring rubric (e.g., 0–5 or 0–10 scale, which would explain non-multiples of 12.5% like 2.5% and 42.5% on the 8-sample "Streaming & Online" subset)? (b) What are the exact input FPS and context window settings for all baseline models in Table 1 (especially native streaming models like JoyAI-VL-Interaction and MOSS-VL-Realtime)?
  3. Since Gemini-3.1-Pro performs the 2-FPS answerability filtering, questions that Gemini fails to answer at 2 FPS are retained, while questions it answers correctly at 2 FPS are discarded. Did the authors evaluate whether this introduces a selection bias that favors or penalizes the Gemini family relative to other model families?
3 comments

We thank the reviewer for recognizing the practical motivation, the annotation pipeline, and the temporal-resolution/history trade-off. The concerns about small subsets, autonomous focus selection, and evaluation transparency are well taken. We address them below and will revise the corresponding claims and protocol descriptions.

W1: Dataset size and the interpretation of small subsets

FastBench's final scale and its capability and temporal-scope distributions emerge from broad candidate collection followed by our quality and high-dynamic-relevance filters. The pipeline selects 306 carefully verified QAs across eight domains from 16,702 candidates, prioritizing visual clarity and dependence on high-dynamic information. This selective curation supports the benchmark's purpose: diagnosing whether streaming models can capture brief temporal evidence and retain it for subsequent questions.

The Streaming & Online Detection subset contains 8 QAs (2.61%), while Backward and Instant contain 65 and 73 QAs, respectively. These category-level results provide descriptive diagnostics of model behavior on distinct streaming tasks. To express the observed improvement precisely, we will revise “+37.5% surge” to state that the mean normalized score increases from 2.5 to 40.0, a gain of 37.5 percentage points on the eight-QA Streaming & Online Detection subset. As detailed in Q2, each answer receives a graded open-ended score, the reported gain therefore reflects changes in answer quality across the subset. We will report category counts alongside these scores to make the basis of each comparison explicit.

W2 / Q1: Autonomous focus, causal delay, and Backward questions

We agree with the reviewer's comparison: autonomous ProactiveFrame does not outperform the cited higher-FPS baselines in overall score. Qwen3-VL-8B achieves 38.3, compared with 38.6 at 4 FPS, 40.5 at 8 FPS, 44.6 at 24 FPS, and 42.1 at 30 FPS; Gemini-3.5-Flash achieves 52.2 versus 55.3 at 30 FPS. The demonstrated gains are relative to the respective 2-FPS baselines. We will state this scope explicitly and characterize ProactiveFrame as a training-free baseline for studying observation control.

How often is focus triggered, and how well does it overlap the evidence? We audited the saved focus logs. Among clips eligible for the interval analysis, Qwen emits at least one Focus_Start in 250/261 clips (95.8%), and Gemini in 276/278 clips (99.3%). Their mean numbers of emitted start commands are 4.61 and 5.65 per clip, respectively.

For predicted focus intervals P, annotated focus intervals G, and intersection duration I, we measure precision as I/|P|, coverage/recall as I/|G|, IoU as I/|P ∪ G|, and F1 as 2I/(|P|+|G|). Metrics are computed per clip and then macro-averaged; all values below are percentages.

ModelEligible clipsIoUCoverage / recallPrecisionF1
Qwen3-VL-8B26119.3835.5328.3926.36
Gemini-3.5-Flash27829.9361.0535.5540.63

Both predicted and GT intervals are clipped to a 10-second window starting at the GT start for Forward, or ending at the GT end for Backward/Instant.

Why can Backward questions benefit? The focus policy operates before a question arrives. The prompt explicitly asks the model to monitor the stream even without an active question and request denser observations when an unfolding event appears likely to require fine temporal detail. For a Backward question, any useful dense evidence must therefore have been captured proactively before that question and remain available when it is answered. The +4.6 and +5.2 point Backward gains over 2 FPS are compatible with this mechanism.

This is harder than the Forward setting, where a known question can direct attention to a target event. We provide the following descriptive comparison under the same analysis procedure:

Temporal-scope groupEligible clipsIoUCoverage / recallPrecisionF1
Forward15634.6972.5538.2945.99
Backward6425.0142.4035.4335.96
Instant6123.3151.1529.7232.35

All four Backward metrics are below Forward, consistent with the difficulty of choosing relevant observations without knowing the future query.

The causal-delay limitation is real. A command emitted after processing a one-second chunk changes sampling only for subsequent chunks. If a brief event ends before focus starts, the policy cannot retrospectively obtain its high-FPS evidence. This applies to an unpredictable one-second spark and to evidence already missed before Backward/Instant queries. Benefits require anticipatory cues, an event that continues into later chunks, or focus already being active. We will explicitly discuss this failure mode.

Finally, annotation-guided focus raises Gemini's overall score to 57.5, compared with 52.2 for autonomous focus and 55.3 for Full 30 FPS. This supports the potential of allocating fine temporal observations to relevant intervals while retaining sparse history. It is an annotation-assisted reference, not a deployable policy or a proven optimal allocation. The benefit is also backbone-dependent: Qwen's oracle score of 40.7 remains below Full 30 FPS (42.1). We will distinguish the potential of selective observation from the limitations of the present autonomous controller.

Q2: Judge, scoring rubric, FPS, and context settings

(a) Open-ended scoring. We follow PhoStream's open-ended evaluation framework. The specified evaluation launcher selects Qwen3-235B-A22B-2507 as the judge. It receives the question, model answer, and reference answer, and is instructed to return an integer score from 0 to 5:

ScoreRubric
5Fully accurate and complete
4Correct and sufficient
3Partially relevant
2Tangential/speculative
1Factually incorrect
0No attempt/off-topic

The rubric accepts coherent, factually plausible answers without requiring every detail or the exact wording of the reference. Each eligible answer's score is multiplied by 20 and then averaged across QAs. Consequently, these percentages are normalized mean scores, not binary accuracy. On an eight-QA subset, one judge-point change on one answer changes the mean by 20/8 = 2.5 percentage points, directly explaining values such as 2.5 and 42.5.

(b) Sampling and context. The common baseline launch configuration specifies a maximum context budget of 248,832,000 pixels. In the shared video-context path, this retains approximately 60 seconds at 2 FPS. The paper's pixel-budget specification, 248,832,000 = 120 × 1920 × 1080, is a 120-frame-at-1080p equivalent. We will distinguish frame caps, pixel budgets, and native history policies, and qualify comparisons accordingly rather than describe all Table 1 rows as strictly context-matched.

Q3: Gemini-based filtering and possible selection bias

The 2-FPS answerability filter serves FastBench's central requirement: questions should depend on high-dynamic visual evidence that sparse temporal sampling can miss. Questions that Gemini-3.1-Pro answers correctly at 2 FPS are excluded because they can already be resolved under the low-FPS setting and therefore do not meet this selection criterion.

The potential model-dependent bias arises among the retained candidates: a question that Gemini cannot answer at 2 FPS might still be answerable by another model from the same sparse observations. Our final human quality check specifically re-examines whether each question requires fine-grained temporal evidence, including whether the answer can instead be obtained from sparse frames or static visual cues. Candidates that do not satisfy this high-dynamic requirement are excluded at this stage. This human review provides an additional check on the relevance of retained questions to the benchmark's objective. We will make these human quality-control criteria explicit in the revised construction description, particularly the requirement to verify dependence on high-dynamic evidence in addition to answer correctness and visual clarity.

We thank the reviewer again for the thoughtful feedback and look forward to any further comments on our clarifications and additional results.

Reviewer_JrzkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces FastBench, a streaming video QA benchmark targeting to evaluate streaming VLMs on high-dynamic real-world video streams. Candidate QA pairs are generated by a VLM from short, high-frame-rate video clips and filtered by the question which can be answered at 2FPS. The remaining QAs are verified or corrected against SAM3 masks and CoTracker3 trajectories. Experiments with 10 models show that the best model reaches 50.7%, that denser sampling helps Qwen3-VL-8B and the proposed ProactiveFrame improves over 2 FPS sampling.

Critical strengths
  • The problem is well motivated. Streaming VLMs operate at low FPS under bounded context, and the resulting failure on fast events is real and under-measured by existing streaming benchmarks.
  • The low-FPS answerability filter is a novel and interesting way to target frame-rate-dependent questions.
  • The FPS sweep under a fixed pixel budget cleanly illustrates the granularity–history trade-off.
Critical weaknesses
  • The benchmark is relatively small (only 306 QAs), and no statistical uncertainty is reported. The results are single runs scored by an LLM judge, which further lacks of confidence intervals.
  • The construction pipeline might introduce bias into the evaluation for Gemini models.
  • Missing larger open source models' performance in the Table 1. It is more improtant due to the biased dataset filtering process.
  • ProactiveFrame does not outperform the simplest dense baseline. Given the average video length, the authors need to further show the significance of proactiveframe.
  • Key dataset statistics are missing like the vidoe duration, distribution of QAs.
  • Novelty relative to prior work is incremental. MotionBench studies fine-grained motion with high-FPS inputs, and PhoStream supplies the streaming protocol that FastBench adopts.
Questions for authors

Refer to weakness, and particularly need to address the bias selection issue and the dataset statistics.

3 comments

We thank the reviewer for recognizing the motivation of FastBench, the low-FPS answerability filter, and the controlled FPS sweep. We address the concerns below with additional model evaluations, repeated judging, uncertainty estimates, and a clearer account of the annotation pipeline and the role of ProactiveFrame.

W1: Benchmark size and statistical uncertainty

FastBench is designed as a focused diagnostic of high-dynamic streaming perception. Starting from broad candidate collection, our quality and high-dynamic-relevance filters select 306 carefully verified QAs over 300 clips across eight domains from 16,702 candidates. The final scale reflects the requirement that retained questions have verified answers, clear visual evidence, and dependence on high-dynamic information. The category distribution likewise emerges from these selection criteria. To complement this curation with quantitative evidence on evaluation reliability, we provide separate analyses of judge variability and finite-sample uncertainty below.

To quantify judge variability, we re-scored the same saved Qwen3-VL-8B responses on all 306 QAs ten times, without rerunning inference:

Repeated LLM judgingOverall score (%)
Mean ± sample standard deviation32.70 ± 0.31
Minimum–maximum32.03–33.07

The standard deviation is in percentage points. These results show that repeated judging produces stable aggregate scores for this fixed response set. We will report that analysis to give readers a quantitative basis for interpreting the benchmark scores.

W2: Potential bias from Gemini-based dataset construction

We appreciate this concern and acknowledge that our description did not make the answer-correction process sufficiently clear. Gemini-3.1-Pro serves as an offline annotator: it examines short, native-frame-rate clips and can use SAM3 masks and CoTracker3 trajectories to verify and correct its initial answers. A candidate with an initially incorrect answer can therefore be retained after correction. The subsequent human quality-control stage also checks and corrects answers against the video. Thus, retention does not require Gemini's original, unaided answer to have been correct.

This annotation setting provides short clips, visual-expert assistance, and human review, whereas evaluated models process the longer videos causally under bounded context. Offline annotation success therefore does not imply that a Gemini model can solve the corresponding streaming task. We will explicitly describe the distinction between initial generation, expert-assisted correction, and final human verification.

W3: Larger open-source models and an updated closed-source baseline

We have evaluated four additional open-source models and GPT-6-Astra on all 306 QAs under their default streaming setting. We include the original Qwen3-VL-8B and Gemini-3.5-Flash results for reference. Scores use the paper's percentage scale.

ModelForwardInstantBackwardOverall
Qwen3-VL-8B (original)33.934.528.332.9
Qwen3-VL-27B (new)49.246.046.247.8
Kimi-K3 (new)51.251.548.350.7
MiniMax-3.1-Flash (new)49.346.847.748.4
GLM-5.3 (new)49.054.248.950.3
Gemini-3.5-Flash (original)52.748.547.750.7
GPT-6-Astra (new, closed-source)53.751.553.253.1

All four added open-source models exceed the strongest original open-source baseline, JoyAI-VL-Interaction at 44.1%. Kimi-K3 matches Gemini-3.5-Flash's overall score to the reported precision, while GPT-6-Astra achieves a higher point estimate. These results show that strong performance is not confined to the annotator's model family and that the benchmark remains challenging for stronger models. We will expand Table 1 and update the claim about the best-performing evaluated model accordingly, without treating these rankings as proof that construction bias has been eliminated.

W4: What ProactiveFrame demonstrates beyond dense sampling

We agree that autonomous ProactiveFrame does not outperform the strongest dense baseline: Qwen3-VL-8B obtains 38.3%, versus 44.6% at 24 FPS, and Gemini-3.5-Flash obtains 52.2%, versus 55.3% at Full FPS. We will clarify its role as a simple, training-free baseline for testing whether existing VLMs can control their temporal observations and balance fine-grained perception with retained history.

The average clip duration of 45.29 seconds does not make this trade-off negligible. In the paper's illustrative 1080p setting, a context capacity equivalent to 120 frames covers 60 seconds at 2 FPS but only 4 seconds at 30 FPS. Consequently, even a clip shorter than five seconds can require selective retention when a brief event must be recalled later. The actual retained duration depends on frame resolution and the sampling policy.

To examine autonomous focus selection, we compare model-selected focus intervals with annotated focus intervals. The following metrics are macro-averaged and expressed as percentages; coverage is temporal recall, and precision is the fraction of predicted focus duration overlapping ground truth.

ModelIoUCoveragePrecisionF1
Qwen3-VL-8B19.3835.5328.3926.36
Gemini-3.5-Flash29.9361.0535.5540.63

Gemini-3.5-Flash selects intervals with higher overlap on all four metrics, although its precision and coverage still leave substantial room for improvement. Its annotation-guided comparison further illustrates the opportunity:

Gemini-3.5-Flash sampling policyOverallBackward
Uniform 2 FPS50.747.7
Autonomous ProactiveFrame52.252.9
Full FPS (30 FPS)55.349.5
Oracle-guided mixed FPS57.555.4

Oracle-guided mixed FPS attains the highest overall score among these settings: +5.3 points over autonomous ProactiveFrame and +2.2 over Full FPS. On Backward questions, which require retaining previously observed evidence, its advantage over Full FPS is 5.9 points. These results support the value of allocating dense observations to relevant intervals while retaining sparse history, and motivate better autonomous decisions about when to focus. The benefit also depends on the backbone: Qwen3-VL-8B's oracle score of 40.7% remains below its Full-FPS score of 42.1%. We will make this scope explicit.

W5: Video duration and QA distributions

Thank you for pointing out that these statistics should be easier to locate. Appendix A.1 and Figure A.1 provide the temporal statistics, while Section 3.2 and Figure 2(b) summarize capability and domain distributions. We will bring the key figures into the main dataset description and provide clearer cross-references:

StatisticValue
Video clips / QA pairs300 / 306
QAs per clip294 clips with one QA; 6 with two
Total video duration3.77 hours
Mean clip duration; range45.29 s; 7.32–79.68 s
Mean query timestamp; range19.34 s; 0–74 s
Forward / Instant / Backward QAs168 (54.90%) / 73 (23.86%) / 65 (21.24%)
Coverage8 domains and 6 capability categories

W6: Novelty relative to MotionBench and PhoStream

We agree that MotionBench establishes the importance of fine-grained motion understanding and that our streaming protocol builds on PhoStream. FastBench's contribution is to operationalize a further evaluation requirement: capturing brief, frame-rate-dependent evidence while preserving the historical context needed for causal streaming QA under a bounded visual budget.

The FPS sweep and oracle-guided experiment make this requirement measurable. For Gemini-3.5-Flash, selective oracle guidance reaches 57.5%, compared with 50.7% at 2 FPS and 55.3% at Full FPS, with a particularly clear difference on Backward questions. This supports evaluating when to acquire dense observations and which evidence to retain, alongside recognition from high-FPS inputs. We will sharpen this contribution statement and position ProactiveFrame as an initial diagnostic baseline, while explicitly crediting the motion-understanding and streaming-protocol foundations of prior work.

We thank the reviewer again for the thoughtful feedback and look forward to any further comments on our clarifications and additional results.

Reviewer_98HzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FastBench, a benchmark for evaluating the perception capability of Streaming VLMs in highly dynamic video scenarios, containing a dataset of 306 QA pairs. Moreover, the paper proposes a training-free method, ProactiveFrame, which balances highly dynamic perception and historical information retention by dynamically adjusting the FPS and using a two-level sliding window. Experiments show that increasing the FPS can improve model performance, but existing models still have shortcomings in autonomously selecting appropriate sampling timing.

Critical strengths
  1. The motivation behind proposing this benchmark in the paper is sound. Existing streaming video understanding benchmarks do not fully consider evaluation environments under highly dynamic conditions. FastBench addresses this issue to some extent.
Critical weaknesses
  1. The paper hopes that the model can perceive highly dynamic events while also retaining previous historical content, but FastBench does not seem to truly evaluate these two capabilities together. Instead, it divides them into two types of tasks and tests them separately, without fully evaluating the trade-off between the two.
  2. The data scale of FastBench may be relatively small compared with some existing benchmarks, and the data distribution within each task category is also not sufficiently balanced. At the same time, the paper evaluates relatively few existing streaming video understanding models.
Questions for authors

See weaknesses and following questions:

  1. The experimental results show that ProactiveFrame achieves only 38.3% on Qwen3-VL, lower than fixed 4 FPS (38.6%) and 24 FPS (44.6%); on Gemini-3.5-Flash it achieves 52.2%, also lower than fixed 30 FPS (55.3%). ProactiveFrame's adaptive frame rate control mechanism does not seem to achieve the expected effect. Is this due to a model issue or a method issue?
  2. The paper does not evaluate actual inference latency and computational overhead, and also lacks a comparison of different FPS strategies under the same computational cost. Could the authors further evaluate ProactiveFrame's inference speed in streaming scenarios?
2 comments

We thank the reviewer for recognizing the motivation of FastBench and for the constructive suggestions. We clarify how Backward questions jointly test dynamic perception and historical retention, provide two additional streaming baselines, and report further analyses of focus selection and inference cost.

W1: Joint evaluation of dynamic perception and historical retention

Backward questions require both capabilities within the same example, the model must capture a brief, highly dynamic event when it occurs and retain the relevant evidence until a subsequent question is asked. Section 3.1 defines Backward questions by an evidence interval that ends before the query. The example in Figure 2(a) asks whether an F1 tire contacted a track feature earlier in the stream. Missing that brief contact through sparse sampling, or losing its evidence as newer frames enter the context, can both prevent a correct answer. Thus, the temporal categories describe when questions relate to events, they do not separate dynamic perception from memory into independent capabilities.

The Gemini-3.5-Flash results from Tables 1 and 2 illustrate this interaction. Scores are percentages. Oracle-guided mixed FPS is the paper's “Mixed FPS (Focus)” setting, which uses annotated focus intervals to allocate dense observations while preserving sparse historical context.

MethodForwardInstantBackwardOverall
Gemini-3.5-Flash (2 FPS)52.748.547.750.7
Gemini-3.5-Flash Full FPS (30 FPS)56.358.149.555.3
Gemini-3.5-Flash Oracle-guided mixed FPS55.464.155.457.5

On Backward questions, Oracle-guided mixed FPS improves over Full FPS by 5.9 percentage points and over 2 FPS by 7.7 points. Uniformly increasing FPS improves Instant performance by 9.6 points but Backward performance by only 1.8 points. This is consistent with the need to balance temporal detail and historical coverage: dense sampling alone provides a smaller benefit when evidence must remain available for a later question. The oracle comparison supports the value of selective allocation within the context budget. We will make the joint requirement of Backward questions more explicit in the paper, these task scores do not by themselves isolate every source of perception or memory error.

W2: Dataset scale, distribution, and coverage of streaming methods

FastBench is designed to diagnose high-dynamic streaming perception through questions that require fine temporal evidence. Starting from broad candidate collection, our quality and high-dynamic-relevance filters select 306 carefully verified QAs over 300 clips across eight domains from 16,702 candidates. The final scale reflects this selective curation: each retained question is checked for answer correctness, visual clarity, and dependence on high-dynamic information. These criteria directly support the benchmark's goal of exposing failures that sparse temporal observations can miss.

We have additionally evaluated StreamingVLM and SimpleStream on all 306 QA pairs:

MethodForwardInstantBackwardOverall
StreamingVLM28.732.634.230.8
SimpleStream40.629.019.733.4

SimpleStream uses its recent-frame policy adapted to our Qwen3-VL-8B evaluation backend and FastBench protocol. These additional results broaden the streaming-method coverage and show that high-dynamic streaming perception remains challenging for these approaches as well.

Q1: Why does autonomous ProactiveFrame underperform some fixed-FPS settings?

We agree that autonomous ProactiveFrame does not surpass the cited fixed-FPS settings: Qwen3-VL-8B scores 38.3%, versus 38.6% at 4 FPS and 44.6% at 24 FPS, while Gemini-3.5-Flash scores 52.2%, versus 55.3% at Full FPS. The results point to imperfect autonomous selection of focus intervals as an important bottleneck. ProactiveFrame relies on the underlying VLM to decide when to request denser observations from the stream seen so far. A missed or delayed focus request can lose critical evidence, while an unnecessarily long request consumes context that could retain earlier observations.

To examine this bottleneck, we compare model-selected focus intervals with the annotated ground-truth intervals. The following metrics are macro-averaged and reported as percentages. Coverage measures the fraction of the ground-truth focus duration covered by the predicted intervals, precision measures the fraction of predicted focus duration overlapping ground truth.

ModelIoUCoveragePrecisionF1
Qwen3-VL-8B19.3835.5328.3926.36
Gemini-3.5-Flash29.9361.0535.5540.63

Gemini-3.5-Flash achieves higher overlap on all four metrics, but even its coverage and precision leave substantial room for improvement. With annotated focus guidance, its overall score reaches 57.5%, exceeding both autonomous ProactiveFrame (52.2%) and Full FPS (55.3%). This controlled comparison supports the benefit of allocating fine temporal observations to relevant intervals and identifies learning when to focus as a promising direction.

We therefore interpret ProactiveFrame as a training-free baseline that exposes current models' limitations in observation control. The oracle result is an annotation-assisted reference, not a deployable policy or proof of an optimal allocation. It also does not rule out improvements to the prompting or control mechanism itself. We will clarify that the evidence supports better focus selection as an important opportunity, rather than claiming that autonomous adaptation already dominates fixed FPS.

Q2: Context budget, computational overhead, and streaming inference speed

As specified in Section 4.1 (“Context Window Constraints”), we define a unified maximum context budget of 248,832,000 pixels, equivalent to the pixel capacity of 120 frames at 1080p resolution (120 × 1920 × 1080). This is a bound on retained visual context.

To quantify these differences, we profiled Qwen3-VL-8B under four strategies with the same pixel-budget constraint. Each strategy processes the stream from the first second and makes one model call per one-second chunk. The following statistics use a subset of 50 samples

MethodMean context framesMean estimated visual tokensMaximum estimated visual tokens per request
2 FPS42.4612,02234,680
Full FPS181.7449,33186,700
Proactive Focus (ProactiveFrame)89.8625,30258,956
Oracle Focus65.2118,34458,956

ProactiveFrame reduces the mean estimated visual-token load by 48.7% relative to Full FPS. These are per-request context measurements, including retained history, rather than counts of unique frames across an entire video.

We thank the reviewer again for the thoughtful feedback and look forward to any further comments on our clarifications and additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.