acceptodds
Under review as a conference paper at ICLR 2027

ASVBench: Asynchronous Response to Fine-Grained Triggers in Streaming Videos

Abstract

Streaming video understanding is a pivotal technology for bridging Multimodal Large Language Models (MLLMs) with real-world AI hardware, such as AR glasses and embodied robots. However, current benchmarks are largely confined to a simplistic “immediate-query, immediate-answer” scenario. We argue that real-world interactions demand a more advanced capability for **Asynchronous Response**, where a model's output is delayed until specific visual conditions are met within a maximum temporal window. This paradigm fundamentally necessitates **fine-grained understanding**, as the model must vigilantly track the subtle, object-centric details that act as the visual triggers. This critical interplay between asynchronous response and fine-grained triggers has been largely overlooked.Therefore, we propose ASVBench, comprising 1452 videos and 4350 challenging Q&A pairs, as the first benchmark for evaluating Asynchronous Response to Fine-grained Triggers in streaming video.Specifically, we build a dual-axis evaluation framework for achieving this goal:(1) The temporal axis assesses models across three core scenarios: Immediate Q&A (responding to the present visual state), Prospective Q&A (responding when future visual triggers are met), and Continuous Q&A (continuously monitoring if triggers are met).(2) The cognitive axis requires the model to respond to visual triggers at different granularities, including recognition-level, comprehension-level, and reasoning-level triggers.ASVBench evaluation results reveal a significant gap between existing MLLMs and human performance, with accuracy plummeting as asynchronous temporal complexity increases. Detailed diagnostics indicate that current models suffer from micro-feature over-smoothing, memory eviction of latent goals, and impatience-driven premature responses, with early errors cascading in multi-shot streams. These insights expose critical bottlenecks in asynchronous responses to fine-grained triggers, providing a roadmap for next-generation streaming MLLMs.

Primary area:
datasets and benchmarks
Keywords:
Streaming Videos, Asynchronous Response, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y52oOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues for a need to evaluate streaming video models in an asynchronous setting, where given a query, the model's response will be at a delayed time until a visual cue is observed. This paper proposes the ASVBench, a fine-grained understanding of streaming video models in the asynchronous response setting.

The benchmark evaluation is split into three categories: immediate Q&A (responding to the query immediately), prospective Q&A (responding to the query when the visual cue is seen in the future), and continuous Q&A (responding to the query at multiple visual cues that can occur). Furthermore, these three categories are split into different question granularities as recognition-type, comprehension-type, and reasoning-type.

The paper evaluates closes source proprietary MLLMs, open source MLLMs (both adopted for the streaming setting), and specialized streaming video models alongside a human benchmark performance. The results indicate severe limitations in existing streaming models and even limitations of general purpose MLLMs in these tasks, especially in the continuous Q&A setting. The performance degradation reasons in the streaming video models are also briefly analyzed.

Critical strengths
  • The paper proposes an important evaluation criteria for streaming video models, which is based on asynchronous response to queries. Given the nature of streaming videos, this evaluation setting is valid and sound. The proposed benchmark scale is modest with 1500 videos and 4400 question and answer pairs.
  • The benchmark evaluations are conducted on both proprietary and open source MLLMs, in addition to streaming video models. This evaluation makes the comparisons quite extensive highlighting interesting limitations of general purpose MLLMs and existing models specialized for streaming videos.
  • The analysis of the failure cases reveal interesting behaviors of existing models such as over smoothing of spatial features.
Critical weaknesses
  • The paper introduces three dimensions of question types: recognition, comprehension and reasoning. While results for these are presented in Table 2, there is no clear discussion on this. As it stands, this categorization does not add much to the rest of the paper. On this, general purpose MLLMs seem to always perform better in reasoning tasks than comprehension tasks. But specialized streaming video models behave the opposite. Could the authors discuss why this is?
  • Could the authors expand on what the J\mathcal{J} values for the different models reveal? LiveStarPro performance for continuous Q&A is in the range 35-38, but J\mathcal{J} is higher. What is the range of this score?
Questions for authors
  • What is the compression used to adapt general purpose MLLMs as mentioned in line 229: ct=Compress(Vt)c_t = \text{Compress}(\mathcal{V}_t)?
  • Is the x axis of image 4b, time. Please specify this.
  • Could the authors refer to the sections (Table, row numbers) that they are discussing in section 5.2 when discussing the results? This will make it easier to follow.
9 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing that our benchmark provides valid and sound asynchronous evaluation criteria, a substantial scale of 1,452 videos and 4,350 question-answer pairs, extensive comparative evaluations across proprietary, open-source, and streaming models, as well as insightful failure analyses such as spatial feature over-smoothing. Below, we address your concerns in detail.

Response to Critical Weakness 1: Cognitive Dimensions and Performance Divergence

We appreciate the reviewer for highlighting these important nuances. Our three cognitive dimensions serve as a diagnostic tool to pinpoint architectural bottlenecks, revealing that streaming models underperform on Reasoning because aggressive token merging erases the fine-grained micro-features required for multi-step causal deduction. We have expanded our discussion in Section 5.2 Finding 3, highlighted in teal, to detail both aspects.

  • The Diagnostic Value of the Cognitive Dimensions: Categorizing tasks into Recognition, Comprehension, and Reasoning systematically disentangles explicit visual fact extraction from holistic event tracking and multi-step causal deduction. By mapping performance across this axis, ASVBench reveals precisely at which cognitive depth an architecture breaks down, moving beyond single overall accuracy metrics.

  • General vs. Specialized Models on Reasoning vs. Comprehension: General-purpose models such as GPT-5.5 possess strong pre-trained logical reasoning and Chain-of-Thought foundations, allowing them to thrive when fed uncompressed visual frames via our causal sliding window. Conversely, streaming-specialized models employ aggressive spatio-temporal token merging for real-time efficiency. As diagnosed in Section 5.3.1, this compression smooths out high-frequency, object-centric micro-features. Because complex Reasoning tasks require tracking precise physical and causal links over time, losing these micro-features impairs reasoning ability, leaving streaming models competent primarily in global Comprehension.

Response to W2: Values of J\mathcal{J} Score and LiveStarPro Performance

We thank the reviewer for allowing us to clarify the formulation of the Joint Asynchronous Score. The J\mathcal{J} score ranges strictly from 0 to 100 and is globally aggregated across all three temporal modes, explaining why LiveStarPro achieves J=43.27\mathcal{J} = 43.27 despite lower accuracy on the harder Continuous Q&A subset. We have updated the Table 2 caption and Section 5.2 Finding 4, highlighted in teal, to ensure complete clarity.

  • The Range and Meaning of the J\mathcal{J} Score: The theoretical range of J\mathcal{J} is strictly from 0 to 100, jointly measuring semantic correctness and temporal responsiveness while assigning 0 points to premature guesses before visual triggers appear.

  • Resolving the LiveStarPro Performance: LiveStarPro achieves a global J\mathcal{J} score of 43.27 because it performs substantially better on Immediate Q&A at 40.13 to 44.21 percent and Prospective Q&A at 40.91 to 44.36 percent. Aggregating across all temporal modes mathematically pulls its overall J\mathcal{J} score higher than its lowest-performing Continuous Q&A subset of 34.63 to 38.64 percent.

Response to Questions For Authors 1: Compression for General Purpose MLLMs

We thank the reviewer for this detailed methodological question. The Compress function introduces no external frame-dropping or heuristic pruning; it strictly represents the model's native vision encoder and built-in spatial pooling mechanisms. We have explicitly clarified this adaptation in Section 3.2.2, highlighted in teal.

For general-purpose models evaluated via the causal sliding window, we apply zero external streaming frame-dropping or heuristic pruning algorithms. The Compress function in Equation 1 mathematically represents the native vision encoder and built-in spatial pooling mechanisms, which natively compress raw pixel frames within the current window Vt\mathcal{V}_t into visual tokens appended to the KV-Cache. This clean baseline isolates streaming-specific architectural flaws from pure perception deficits.

Reviewer_yJ1ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ASVBench, a benchmark for evaluating asynchronous responses to fine-grained visual triggers in streaming videos. The authors construct 4.3K QA pairs across temporal and cognitive dimensions and evaluate 20 MLLMs. The results reveal substantial limitations in current streaming models, particularly in response timing and fine-grained understanding.

Critical strengths
  1. The paper addresses a timely and important problem in streaming video understanding. The reviewer agrees that prospective and asynchronous responses are essential for real-world applications.

  2. The proposed benchmark provides a useful evaluation of fine-grained visual understanding and response timing. The extensive model comparisons offer interesting insights into current streaming MLLMs.

Critical weaknesses
  1. The authors claim this as the first benchmark for asynchronous responses to fine-grained triggers. However, as shown in Tab. 1, several existing benchmarks already evaluate proactive responses and event timing. The reviewer believes the authors should tone down this claim and more clearly highlight the fundamental distinctions or niches from existing benchmarks.

  2. The inference protocol in section 3.2.1 is unclear. What exactly does the memory state refer to (KV cache, visual tokens, or something else)? While some end-to-end streaming models are trained to generate special tokens (<WAIT>), many other proactive streaming models adopt agentic or modular approaches that control response timing differently. It is unclear how the proposed protocol can be consistently applied to these architectures for a fair evaluation.

  3. The reviewer has concerns about the reliability of the data construction pipeline. The authors rely heavily on frontier MLLMs to generate fine-grained event annotations and ground-truth trigger timestamps. However, accurate temporal grounding involving subtle visual cues and causal reasoning remains challenging even for frontier models. Although human verification is included, the reported agreement does not directly validate timestamp accuracy. Since response timing is central to this benchmark, more rigorous validation of temporal annotations is necessary.

Questions for authors
  1. Overall, the reviewer agrees that the paper addresses important challenges in streaming video understanding. However, the automated construction pipeline largely relies on frontier MLLMs without sufficient validation of temporal grounding reliability, which raises concerns about the benchmark's validity.

  2. As a minor comment, although this task does not necessarily require IRB, the authors should clarify how human experts were involved in the annotation and validation process. Such details are important for establishing the reliability of the benchmark even if ICLR venue do not explitly require such information.

9 comments

We sincerely thank the reviewer for the constructive feedback and the positive comments. We are highly encouraged that you recognize our work addresses a timely and important problem in streaming video understanding, that prospective and asynchronous responses are essential for real-world applications, and that our extensive model comparisons offer valuable insights. Below, we address your concerns in detail.

Response to Critical Weakness 1: Overclaimed First Benchmark and Distinctions

We appreciate the reviewer’s careful comparison with existing datasets. ASVBench is the first benchmark to couple fine-grained, object-centric visual triggers with asynchronous response timing, whereas existing benchmarks evaluate either offline fine-grained perception or macroscopic streaming events. To prevent any ambiguity, we have refined our phrasing in Section 2 Related Work, highlighted in red.

BenchmarkVideos and QAStreamingFine-GrainedAsynchronousContinuous MonitoringTimestamp-AwareEvent-Timing
Video-MME900 and 2.7KNoNoNoNoNoNo
TemporalBench2K and 10KNoYesNoNoNoNo
VideoRefer198 and 1KNoYesNoNoNoNo
StreamingBench900 and 4.5KYesNoNoYesYesNo
OVBench1.5K and 7KYesNoNoYesYesNo
OVO-Bench644 and 2.8KYesNoYesNoYesNo
SVBench1.4K and 50KYesNoNoYesYesNo
TV-Online50K and 200KYesNoYesYesYesNo
OmniPro1.3K and 2.7KYesNoYesYesYesYes
ProactiveBench911 and 911YesNoYesYesYesYes
ASVBench Ours1.5K and 4.4KYesYesYesYesYesYes

As detailed in the comparison table above and Table 1 in the manuscript, existing benchmarks fail to simultaneously couple fine-grained visual tracking with asynchronous timing. Offline benchmarks such as Video-MME, TemporalBench, and VideoRefer assume full video access at inference time, preventing causal streaming evaluation. Early streaming benchmarks like StreamingBench, OVBench, and SVBench focus predominantly on immediate responses rather than trigger-driven interactions. Even recent asynchronous benchmarks including OVO-Bench, TV-Online, OmniPro, and ProactiveBench focus strictly on macroscopic events and lack object-centric micro-feature granularity.

In contrast, ASVBench uniquely integrates micro-feature recognition, comprehension, and reasoning with long-horizon goal retention and precise event-timing evaluation. Real-world AI hardware such as augmented reality glasses and embodied robots must discern subtle micro-features before acting at the exact right moment, making ASVBench a foundational roadmap for next-generation streaming architectures.

Response to Critical Weakness 2: Inference Protocol and Memory State

We appreciate the reviewer for highlighting the need for architectural specificity. The memory state MtM_t is an architecture-specific representation of retained visual context, while WAIT functions as a unified system-level evaluation interface rather than a mandatory generated text token. We have explicitly detailed these engineering implementations in Section 3.2.1, highlighted in red, to ensure complete transparency.

First, the memory state MtM_t is tailored to specific architectures. For general-purpose LLMs, MtM_t mathematically refers to the accumulating KV-Cache managed via prefix caching. For streaming-specialized models, MtM_t refers to their internal visual token memory banks or compressed spatiotemporal states.

Second, WAIT is a unified evaluation interface state rather than a forced generative token. For end-to-end generative models, if the output lacks the definitive answer entity, the evaluation wrapper maps this step to WAIT. For modular or agentic streaming models with a gating mechanism, if the temporal gate decides not to respond at step tt, the script logs it as a WAIT state. This unified interface strictly isolates the timing mechanism across offline and streaming architectures, establishing a fair perceptual upper bound and clear performance boundaries for future streaming model development.

Response to Critical Weakness 3: Temporal Grounding Reliability

We are grateful to the reviewer for highlighting the paramount importance of temporal accuracy in our data construction pipeline. A newly conducted human validation across 150 samples confirms that our automated pipeline achieves a Mean Absolute Error of only 0.42 seconds against human gold standards, a sub-second deviation absorbed by our 1-second inference step size. We have detailed this quantitative validation in Appendix B and Table 3, and referenced it in Section 4.2 Step 7, highlighted in red.

Specifically, we randomly sampled 150 pairs from the final test set covering Immediate, Prospective, and Continuous Q&A modes with 50 samples each. Four independent domain experts blinded to model outputs manually annotated the exact timestamps of the fine-grained visual triggers using a frame-by-frame video player, and we computed the Mean Absolute Error between automated pipeline timestamps and human gold-standard timestamps.

Task ModeSample SizeMAE in Seconds (↓\downarrow)Error under 1s (↑\uparrow)
Immediate Q&A500.3896.0
Prospective Q&A500.4594.0
Continuous Q&A500.4394.0
Overall1500.4294.7

As reported in the validation table above, the overall MAE is only 0.42 seconds, with 94.7 percent of samples exhibiting a temporal error under 1.0 second. This quantitative evidence confirms that the temporal boundaries established by our automated data engine are highly reliable, reinforcing ASVBench as a trustworthy foundation for strict asynchronous evaluation.

Reviewer_xyJTOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASVBench, a benchmark for evaluating asynchronous responses to fine-grained visual triggers in streaming videos, covering three temporal scenarios—immediate Q&A, prospective Q&A, and continuous Q&A—and three cognitive levels: recognition, comprehension, and reasoning.

Critical strengths
  1. The benchmark is well-constructed, comprising 1,452 videos and 4,350 Q&A pairs.
  2. The experiments are extensive, comparing human performance with various MLLMs, including general-purpose and streaming-specialized models.
  3. The paper provides detailed analyses, supported by extensive visualizations of experimental results.
Critical weaknesses
  1. The paper emphasizes fine-grained visual triggers as a key contribution, but it is unclear whether fine-grained visual information is essential for detecting these triggers. Without sufficient evaluation of visually similar but non-matching events, the benchmark may not effectively distinguish fine-grained trigger detection from coarse-grained event recognition.
  2. The asynchronous interaction setting is formulated as delayed question answering rather than persistent instruction following, which may not fully reflect realistic streaming interactions. Users generally expect immediate responses to queries, whereas proactive responses are more naturally triggered by persistent instructions.
  3. Continuous Q&A requires models to provide two different answers to a single question at predefined timestamps, which may not fully reflect realistic continuous monitoring.
  4. The Joint Asynchronous Score assigns zero credit to early responses while allowing exponentially decayed scores for delayed responses, potentially penalizing models that correctly infer answers before the annotated timestamps.
Questions for authors

How does the evaluation handle additional responses or incremental outputs across multiple chunks from streaming models such as MiniCPM-o-4.5?

8 comments

We sincerely thank the reviewer for the highly constructive feedback. We are greatly encouraged that you recognize ASVBench as a well-constructed benchmark covering three temporal scenarios and three cognitive levels, supported by extensive experiments comparing human performance with diverse multimodal models, as well as detailed analyses and visualizations. Below, we address your concerns in detail.

Response to Critical Weakness 1: Essentiality of Fine-Grained Visual Information

We deeply appreciate this insightful critique. Fine-grained visual information is strictly essential in real-world streaming environments because coarse-grained event recognition inevitably triggers severe false positives on visually similar pseudo-triggers. To explicitly prove this necessity, we constructed a new robustness subset named ASVBench-HardNeg comprising 100 carefully selected pseudo-trigger samples, detailed in Appendix C, Table 4, Figure 7, and Figure 8, and referenced in Section 5.2 Finding 1, highlighted in blue.

ModelBase Acc (↑\uparrow)HardNeg Acc (↑\uparrow)Δ\Delta Drop (↑\uparrow)FPR (↓\downarrow)
Gemini-2.5-Pro42.4522.45-20.0058.4
Seed-1.6-vision51.3824.38-27.0063.5
Qwen3-VL 8B44.9619.96-25.0066.2
LiveStarPro 8B34.6312.63-22.0082.3
MiniCPM-o-4.5 9B35.2914.29-21.0079.6

As shown in the table above, our dataset naturally contains long videos with visually similar but semantically different actions. For example, in Egocentric Sports, we used footage of non-target players playing table tennis as pseudo-triggers before the true target comparison frame. In Daily Life scenarios, we used the half-baked state of food which is expanded but not caramelized as a pseudo-trigger. The experimental results demonstrate that existing models suffer from severe false positives when facing these pseudo-triggers, with streaming models like LiveStarPro 8B reaching an 82.3 percent false positive rate. This compellingly proves that relying merely on coarse-grained event recognition causes severe misjudgments in streaming interactions, aligning with the Impatience Bias diagnosed in Section 5.2. By exposing these real-world vulnerabilities, our research establishes a strict standard that guides future streaming models to achieve precise identity verification and state perception in real-world applications.

Response to Critical Weakness 2: Delayed QA vs. Persistent Instruction

We thank the reviewer for raising this thoughtful point regarding realistic streaming interactions. Our Delayed Q&A formulation is mathematically equivalent to persistent instruction following, as models must retain the query as a latent goal and continuously output WAIT until visual conditions are met. We clarify this equivalence in Section 1, highlighted in blue. As illustrated in our Introduction with the smart oven example of notifying the user when the cake is ready, setting a query and requiring the system to autonomously conduct asynchronous monitoring until the visual condition is met is precisely how real-world persistent instructions operate. We mathematically formulated this as maintaining the WAIT state to stringently test instruction memory without decay over a long temporal horizon.

ModelProspective REC (↑\uparrow)Prospective COM (↑\uparrow)Prospective REA (↑\uparrow)Continuous REC (↑\uparrow)Continuous COM (↑\uparrow)Continuous REA (↑\uparrow)
Flash-VStream 7B18.6732.5732.0529.8034.1746.19
VideoChat-Online 4B28.4631.2229.5125.8726.1024.39
StreamChat 7B42.5339.6038.4939.8137.1336.52
TimeChat-Online 7B32.4028.6530.0727.5325.9626.47
MiniCPM-o-4.5 9B43.2940.5438.6737.8035.2934.92
LiveStarPro 8B44.3641.2840.9138.6434.6336.89
VideoChat3 4B39.0638.7230.5927.2024.1325.17

As presented in Table 2 and detailed across all six columns of accuracy in percent above, we evaluated all seven open-source streaming-specialized models strictly under this persistent instruction-following protocol. These experiments reveal that maintaining instruction memory over long horizons remains a severe bottleneck for current architectures. Even top-performing streaming models such as LiveStarPro 8B and MiniCPM-o-4.5 9B drop from 44.36 percent and 43.29 percent on Prospective Recognition down to 34.63 percent and 35.29 percent on Continuous Comprehension, while TimeChat-Online 7B and VideoChat3 4B fall below 28 percent across all Continuous Q&A dimensions. By exposing these critical flaws regarding memory eviction and latent goal decay, ASVBench provides vital diagnostic value for building reliable always-on streaming AI agents.

Response to Critical Weakness 3: Predefined Timestamps in Continuous Q&A

We appreciate the attention of the reviewer to the continuous monitoring setup, and we respectfully take this opportunity to clarify this mechanism. No predefined timestamps are exposed to models during inference; they serve strictly as hidden backend ground truths to evaluate multi-stage continuous tracking in real-world scenarios. We have clarified this inference mechanism in Section 3.2.2 Mode 3, highlighted in blue.

First, models process the causal video stream on the fly with a fixed step size of Δt=1\Delta t=1 second and must autonomously output a WAIT signal during the inter-trigger monitoring phase. The predefined timestamps are strictly hidden ground truths used only by the backend evaluation script to calculate the joint asynchronous score in Equation 4.

Second, continuous monitoring in reality is rarely a one-off event. For instance, in daily life scenarios such as eating a meal, a user may issue a persistent request requiring two sequential answers as the dining process progresses, demanding dynamic answer switching rather than shutting down after the first response. Similarly, as shown in Table 5, tracking how Simon Gauzy adapts his strategy across different stages of a table tennis match requires continuous state tracking. By forcing models to maintain persistent monitoring across sequential visual conditions without premature termination, ASVBench provides vital value for evaluating continuous-perception hardware like augmented reality glasses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.