acceptodds
Withdrawn from ICLR 2027

EviStream: Evidence-Aligned Audio-Visual Decisions over Partially Observable Video Streams

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) could enable assistants that continuously monitor video streams and respond when a user-specified condition is met. Yet recognizing the current scene is insufficient for deciding when to respond: some events depend on earlier observations, while spoken evidence becomes available only after transcription. Under bounded context, a streaming assistant must preserve relevant task progress and determine whether the available evidence supports a response. We introduce EviStream, a training-free framework for evidence-aligned audio-visual decisions over partially observable video streams. Its central idea is to manage observed evidence, persistent task progress, and temporary model judgments separately. A bounded KV cache retains recent visual and transcript evidence, while a temporary inference branch produces compact, evidence-indexed observations. A persistent state tracks established prerequisites and consumed observations to determine whether to wait or trigger a notification. Incremental speech transcripts are incorporated as they become available, and answers are generated from the frozen triggering context. Using frozen Qwen3-VL-32B-Instruct, EviStream achieves a joint timing-and-content score of 25.3 on all 774 StreamArena Pro questions, compared with 4.7 for direct window monitoring with the same backbone. Removing persistent state, audio, or sliding KV reduces the score to 10.3, 22.1, and 14.9, respectively. These results support the effectiveness of coordinating evidence retention and task progress for proactive streaming responses without task-specific fine-tuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video understanding, vision-language models, audio-visual reasoning, evidence alignment, KV cache, proactive response

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tfaNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EviStream, a training-free proactive streaming assistant built on frozen Qwen3-VL and streaming ASR. It keeps a bounded sliding KV cache of recent evidence, decodes grammar-constrained per-frame labels in a temporary branch, and updates a persistent task state with a hand-written reducer.

Critical strengths
  1. The problem is well motivated, and the different clocks are carefully separated.
Critical weaknesses
  1. Generality is a major concern. The task compiler relies on hand-written rules that look tuned to StreamArena Pro wording (e.g., "next" plus "product", "actual eating rather than food preparation", Appendix B), and all results are on the same 774 questions with one backbone. I suggest the authors explain how the rules were built and evaluate on a held-out split or another proactive benchmark (e.g., OVO-Bench, StreamingBench).
  2. The ablations do not support the claim that persistent task state is what matters. Only 27 questions are change/reappearance, where the full method gets 3 vs. 1 without S. Most of the drop without S comes from single-label event questions, and the no-notification rate jumps from 27.0% to 63.3% without explanation. I suggest the authors separate the cursor reset from the progress reset.
  3. Weak experimental comparison. Window-Qwen gives no notification on 85.1% of questions, JoyAI scores 0.0, and the control responses seem to be scored differently from EviStream (Appendix C.1). A same-backbone baseline with the same compiled prompt and the same scoring pipeline for all rows is needed. The paper should also discuss closely related proactive streaming methods such as Dispider [*1] (which also separates perception, decision and reaction) and VideoLLM-online [*2].

[*1] Rui Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR 2025. [*2] Joya Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024.

Questions for authors

As in weakness.

Reviewer_RcLvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EviStream, a training-free framework for proactive audio-visual streaming understanding. The method separates retained evidence, temporary VLM observations, and persistent task state, using explicit state transitions to determine when to trigger notifications. It also incorporates incremental speech transcripts and KV-cache reuse for bounded-context monitoring. On StreamArena Pro, EviStream achieves a joint timing-and-content score of 25.3, compared with 4.7 for a direct window-monitoring baseline using the same backbone.

Critical strengths
  1. The paper present a well-motivated and practical framework for proactive streaming video understanding. This decoupling of retained evidence, transient VLM judgments, and persistent task state leads to a clean and interpretable mechanism to maintain historical event conditions and make response decisions.
  2. The method demonstrates substantial empirical improvements on a challenging benchmark. EviStream achieves 25.3 on all 774 StreamArena Pro questions, compared with 4.7 for the same-backbone window-monitoring baseline. The component ablations also show meaningful contributions from state management and audio integration, particularly on speech-dependent questions.
Critical weaknesses
  1. The contribution of explicit historical state tracking is not sufficiently isolated. The no-state ablation removes both semantic progress and the consumed-frame cursor, making it unclear which component drives the substantial performance improvement. Moreover, the method succeeds on only 3 of the 27 change/reappearance questions, where historical state tracking should be particularly important. This weakens the experimental evidence for one of the paper's central contributions.

  2. The evaluation does not fully reflect real-time notification performance. The official metric uses the cited frame timestamp rather than the actual notification time, so a response may receive timing credit even if the notification is delayed. Although the authors record notification timestamps, they do not report the corresponding timing accuracy or latency distributions. This makes it difficult to assess whether the improvements translate into timely proactive responses.

Questions for authors
  1. Since the no-state ablation removes both semantic progress and the consumed-frame cursor, can the authors clarify which mechanism primarily contributes to the 15-point performance drop, based on the existing results or execution logs?

  2. Can the authors report notification-time accuracy and latency statistics using the recorded local notification timestamps, rather than only the cited frame timestamps, to clarify whether the improvements translate into timely responses?

  3. For the 27 change/reappearance questions, can the authors provide a breakdown of the main failure modes and clarify whether the successful cases actually require tracking historical prerequisites beyond the retained visual window?

Reviewer_XkqrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

EviStream is a training-free framework for proactive audio-visual monitoring. It combines a bounded evidence cache, persistent task state, and a frozen VLM that produces observation labels for a rule-based decision module. Speech transcripts are incorporated as they become available, and answers are generated from the triggering context. On 774 StreamArena Pro questions, the method scores 25.3, compared with 4.7 for a same-backbone window baseline. Ablations examine persistent state, audio, and KV reuse.

Critical strengths

The framework provides an explicit interface between model predictions and notification decisions. Indexed observations, deterministic transition rules, and persistent consumption records make the process more inspectable. The substantial improvement over a same-backbone baseline also suggests that the complete procedure merits further investigation.

Critical weaknesses
  1. The evaluation does not adequately validate timely, evidence-supported decisions. The reported score uses the cited frame’s source time, although its label can depend on later observations and transcripts available at dispatch. Answer generation similarly uses the entire dispatch context. A successful score therefore does not establish that sufficient evidence was available at the credited frame time or that the notification was delivered promptly. The paper acknowledges this distinction but provides no aggregate notification-delay analysis or assessment of retrospective labeling. This leaves a central aspect of the proposed evidence alignment unvalidated.

  2. The experiments do not convincingly establish the central contribution of persistent task state. Removing state also resets the consumed-frame cursor, changing repeated interpretation and potentially processing time. KV recomputation likewise changes both historical representations and computational workload. These comparisons leave the causes of improvement unclear. On the directly relevant change/reappearance subset, the complete method succeeds on only 3/27 questions, versus 1/27 without state. Long-horizon performance is also weak and does not improve over the state-removal variant. Consequently, the evidence for the intended sustained-monitoring capability remains limited.

Questions for authors
  • What are the joint accuracy and delay distributions when timing is measured at notification? How often do credited predictions rely on evidence arriving after the cited frame?

  • What happens when semantic task progress is removed while the consumed-frame cursor is preserved? This control is needed to isolate the contribution of persistent progress.

  • Please clarify the MiniCPM-o-4.5 results in Table 4 and provide raw correct counts and denominators. For example, 6.3% on 259 questions cannot arise from an integer success count rounded to one decimal place under the stated binary metric.

Reviewer_zF4cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EviStream, a training-free framework for proactive responses over audio-visual video streams. It separates three things: a bounded sliding KV cache of recent frames and ASR transcripts, a disposable inference branch in which a frozen Qwen3-VL-32B emits grammar-constrained labels for each new frame, and a persistent task state that a deterministic reducer updates under hand-written event, change, and reappearance rules to decide WAIT or ALERT. Speech is admitted only once its streaming Paraformer transcript is available, and answers are generated from a frozen snapshot of the triggering context. On all 774 StreamArena Pro questions, EviStream scores 25.3, compared with 4.7 for a same-backbone window-monitoring baseline. Removing the persistent state, audio, or sliding KV lowers the score to 10.3, 22.1, and 14.9, respectively.

Critical strengths
  1. The problem is clearly defined. The trade-off between long-term memory and real-time perception is a problem of practical significance.

  2. The architecture is clearly designed and interpretable.

  3. The method is training-free. It directly uses a frozen Qwen3-VL-32B, which lowers the barrier to swapping backbones or deploying the system.

Critical weaknesses
  1. Poor long-horizon performance. On the L3 subset, the method scores only 7.5, so the paper’s core claim lacks sufficient experimental support.

  2. Limited experimental scope. The evaluation uses only a single backbone and a single benchmark, and there are no results on other benchmarks such as OVOtiming or ProactiveVQA.

  3. No latency or computational cost measurements. The design is largely motivated by real-time streaming, yet the paper reports no end-to-end latency, throughput, or GPU cost, so it is impossible to judge whether the system can actually run in real time.

  4. Limited applicability beyond query-first settings. The method appears to target the setting where the question is known in advance. It seems inapplicable to multi-turn dialogue or to cases where the question arrives after the relevant events have already occurred.

  5. Limited novelty. The method mainly combines existing techniques.

  6. Insufficient ablation studies. The ablations do not adequately isolate the contribution of each component.

Questions for authors
  1. Applicability beyond query-first settings. The framework assumes the question is registered before the relevant events occur. How would it handle multi-turn interaction, or questions that arrive after the relevant evidence has already been evicted?

  2. Consistency of content evaluation. Section 4.1 states that EviStream and its ablations are judged by the official Gemini evaluator, while Appendix C.1 describes the controls as “exact reference-phrase matches accepted automatically and other responses reviewed individually.” Were all systems in Table 4 judged with the same procedure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.