- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues for a need to evaluate streaming video models in an asynchronous setting, where given a query, the model's response will be at a delayed time until a visual cue is observed. This paper proposes the ASVBench, a fine-grained understanding of streaming video models in the asynchronous response setting.
The benchmark evaluation is split into three categories: immediate Q&A (responding to the query immediately), prospective Q&A (responding to the query when the visual cue is seen in the future), and continuous Q&A (responding to the query at multiple visual cues that can occur). Furthermore, these three categories are split into different question granularities as recognition-type, comprehension-type, and reasoning-type.
The paper evaluates closes source proprietary MLLMs, open source MLLMs (both adopted for the streaming setting), and specialized streaming video models alongside a human benchmark performance. The results indicate severe limitations in existing streaming models and even limitations of general purpose MLLMs in these tasks, especially in the continuous Q&A setting. The performance degradation reasons in the streaming video models are also briefly analyzed.
Critical strengths
- The paper proposes an important evaluation criteria for streaming video models, which is based on asynchronous response to queries. Given the nature of streaming videos, this evaluation setting is valid and sound. The proposed benchmark scale is modest with 1500 videos and 4400 question and answer pairs.
- The benchmark evaluations are conducted on both proprietary and open source MLLMs, in addition to streaming video models. This evaluation makes the comparisons quite extensive highlighting interesting limitations of general purpose MLLMs and existing models specialized for streaming videos.
- The analysis of the failure cases reveal interesting behaviors of existing models such as over smoothing of spatial features.
Critical weaknesses
- The paper introduces three dimensions of question types: recognition, comprehension and reasoning. While results for these are presented in Table 2, there is no clear discussion on this. As it stands, this categorization does not add much to the rest of the paper. On this, general purpose MLLMs seem to always perform better in reasoning tasks than comprehension tasks. But specialized streaming video models behave the opposite. Could the authors discuss why this is?
- Could the authors expand on what the values for the different models reveal? LiveStarPro performance for continuous Q&A is in the range 35-38, but is higher. What is the range of this score?
Questions for authors
- What is the compression used to adapt general purpose MLLMs as mentioned in line 229: ?
- Is the x axis of image 4b, time. Please specify this.
- Could the authors refer to the sections (Table, row numbers) that they are discussing in section 5.2 when discussing the results? This will make it easier to follow.
9 comments
Response to W2: Values of Score and LiveStarPro Performance
We thank the reviewer for allowing us to clarify the formulation of the Joint Asynchronous Score. The score ranges strictly from 0 to 100 and is globally aggregated across all three temporal modes, explaining why LiveStarPro achieves despite lower accuracy on the harder Continuous Q&A subset. We have updated the Table 2 caption and Section 5.2 Finding 4, highlighted in teal, to ensure complete clarity.
-
The Range and Meaning of the Score: The theoretical range of is strictly from 0 to 100, jointly measuring semantic correctness and temporal responsiveness while assigning 0 points to premature guesses before visual triggers appear.
-
Resolving the LiveStarPro Performance: LiveStarPro achieves a global score of 43.27 because it performs substantially better on Immediate Q&A at 40.13 to 44.21 percent and Prospective Q&A at 40.91 to 44.36 percent. Aggregating across all temporal modes mathematically pulls its overall score higher than its lowest-performing Continuous Q&A subset of 34.63 to 38.64 percent.
Response to Questions For Authors 1: Compression for General Purpose MLLMs
We thank the reviewer for this detailed methodological question. The Compress function introduces no external frame-dropping or heuristic pruning; it strictly represents the model's native vision encoder and built-in spatial pooling mechanisms. We have explicitly clarified this adaptation in Section 3.2.2, highlighted in teal.
For general-purpose models evaluated via the causal sliding window, we apply zero external streaming frame-dropping or heuristic pruning algorithms. The Compress function in Equation 1 mathematically represents the native vision encoder and built-in spatial pooling mechanisms, which natively compress raw pixel frames within the current window into visual tokens appended to the KV-Cache. This clean baseline isolates streaming-specific architectural flaws from pure perception deficits.
We sincerely thank the reviewer for the constructive feedback and for recognizing that our benchmark provides valid and sound asynchronous evaluation criteria, a substantial scale of 1,452 videos and 4,350 question-answer pairs, extensive comparative evaluations across proprietary, open-source, and streaming models, as well as insightful failure analyses such as spatial feature over-smoothing. Below, we address your concerns in detail.
We appreciate the reviewer for highlighting these important nuances. Our three cognitive dimensions serve as a diagnostic tool to pinpoint architectural bottlenecks, revealing that streaming models underperform on Reasoning because aggressive token merging erases the fine-grained micro-features required for multi-step causal deduction. We have expanded our discussion in
Section 5.2 Finding 3, highlighted in teal, to detail both aspects.The Diagnostic Value of the Cognitive Dimensions: Categorizing tasks into Recognition, Comprehension, and Reasoning systematically disentangles explicit visual fact extraction from holistic event tracking and multi-step causal deduction. By mapping performance across this axis, ASVBench reveals precisely at which cognitive depth an architecture breaks down, moving beyond single overall accuracy metrics.
General vs. Specialized Models on Reasoning vs. Comprehension: General-purpose models such as GPT-5.5 possess strong pre-trained logical reasoning and Chain-of-Thought foundations, allowing them to thrive when fed uncompressed visual frames via our causal sliding window. Conversely, streaming-specialized models employ aggressive spatio-temporal token merging for real-time efficiency. As diagnosed in
Section 5.3.1, this compression smooths out high-frequency, object-centric micro-features. Because complex Reasoning tasks require tracking precise physical and causal links over time, losing these micro-features impairs reasoning ability, leaving streaming models competent primarily in global Comprehension.