acceptodds
Under review as a conference paper at ICLR 2027

Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

Abstract

Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce __Omni-InteractScope__, a benchmark of 1,500 human-annotated, cross-reviewed interactions with 1,814 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) __Timely tool invocation remains unreliable__: all 12 evaluated models produce an on-time first response for fewer than 40% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 31.1%. (2) __Query-free reminders remain poorly covered__: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0%. (3) __End-to-end guidance coverage remains limited__: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.

Primary area:
datasets and benchmarks
Keywords:
Omnimodal Interaction, Multimodal Large Language Models, Audio-Visual Understanding, Streaming Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NvxcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Omni-InteractScope, a benchmark aimed to measure how models can intervene on the correct timing based on stream evidence and user need. The dataset was created using online videos and human annotations and presents a diversity of tasks and scene distributions, from tool invocation to nonverbal and impairment assistance. 12 models are evaluated using this benchmark. The paper presents the following contributions: benchmark with multiple evaluation subtasks and evaluation on 12 multimodal models.

Critical strengths
  • The dataset is created using human annotations.
  • The dataset gathers a diverse range of tasks and scene distributions
  • The evaluation is done with a broad range of models, including open/closed source models, different families and streaming models
Critical weaknesses
  • The paper is not easy to follow and lacks clarity. This includes no reference to the number of annotators, how were the annotators selected. If being human annotated is part of the contributions, then this needs to be as much clear as possible.
  • The dataset construction and justification on the visual data used is vague or non existent.
  • The analysis is majorly focused on the main results and there is just a brief analysis beyond that.
  • There is no ablation study to understand if non-omni models perform better than omni because of the ASR model used.
  • Most tasks (T2-T8) are evaluated using a closed source model (gpt-4o) without cross human or model correlation.
  • There is no claim regarding the availability of the dataset which makes the major contribution weaker.
Questions for authors
  • Will the videos be part of the data? If not are they accessible to everyone?
  • What was the selection method for the sources of the videos?
  • How was AI used for data validation (mentioned in the AI Statement) before final human verification?
  • Can you do the evaluation with humans or other model and measure the correlation with gpt-4o? It is fundamental to understand if the results presented are biased to the model or if the results generalize with other judges.
  • Which ASR was used?

Suggestions:

  • Task numbering should follow task clustering so this makes it easier to follow and understand by looking at the text and figure in the same order.
  • Line 263 misses a bold on T1.
  • Referencing a task of the dataset as “Our T1” (L.149) confounds more then helps.
  • Digging more into analysis such as section 5 would make the paper much stronger.
  • If a generic model + ASR is better then this should be more relevant on the results and analysis.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Nvxc

We appreciate your time and effort in reviewing our manuscript. Below are our detailed point-by-point responses.

Responses to Weaknesses

W1: Annotator details and presentation clarity

We have updated Section 3.3 and Appendix A.3 to explicitly document our annotation team and quality-control workflow, and we have reorganized Section 3.2 to follow a strict T1 to T8 sequence (matching Figure 2 and Table 4) for better readability:

  • Annotator selection and training: Our annotation team comprised 40 undergraduate and master's students with prior experience in multimodal data labeling. Candidates were selected after completing benchmark-specific training and qualification trials on task taxonomy and temporal boundary marking.
  • Two-stage verification: Every interaction was first annotated for user context, trigger evidence, reference response, and valid time window, and then independently cross-reviewed by separate human reviewers to verify both semantic accuracy and second-level temporal alignment.

W2: Dataset construction and visual sources

We have expanded Appendix A.3, complementing Section 3.3, to detail the video acquisition, screening, and annotation pipeline across our 1,128 continuous videos:

  1. Explicit-request tasks (T1 to T7): We collected 920 online videos from Douyin, Xiaohongshu, and Bilibili by querying diverse daily-life, workflow, and digital-screen categories. Candidate clips were manually screened to filter out static, low-resolution, or heavily edited montage videos, retaining continuous streams with clear causal state transitions and natural audiovisual triggers.
  2. Impairment assistance (T8): To avoid web-data bias and reflect authentic assistive needs, we recorded 208 first-person (egocentric) videos across realistic indoor and outdoor scenarios encountered by individuals with visual or hearing impairments (e.g., street navigation, reading signage, locating household items, and monitoring environmental alerts across T8-A/B/C).

W3: Depth of diagnostic analysis

Beyond the main benchmark rankings (Section 4.2), Section 5 and Appendix D (Tables 8 to 17) provide extensive controlled diagnostics into why current Omni-LLMs fail:

  • Scene grounding vs. request priors (Section 5.1, Appendix D.1): Paired no-scene ablations, event-shifted/absent counterfactuals, and fixed-delay controls demonstrate that models heavily exploit text priors in tool invocation while the two unchanged-request cases track the five-second event shift but fail the joint occurrence, timing, and tool-choice criteria.
  • Response history, frequency, and scheduling (Section 5.2, Appendix D.2): Controlled runs isolate how conditioning on response history cuts output volume by 58.1% while preserving reminder recall (22/122), and how post-hoc response caps (K∈{1,3,5,10}K \in \{1, 3, 5, 10\}) affect coverage. Separate controls examine event cues under autonomous vs. scheduled timing across our core baselines.
  • Sequence continuity, sensitivity, and failure modes (Section 5.3, Appendix D.3 to D.4): We analyze multi-step completion via permutation tests (Figure 6), decision-schedule sensitivity (Table 14), temporal tolerance (Table 15), premature vs. content failures (Table 16), and audio modality ablations (Table 17), alongside paired qualitative case studies in Appendix E.

We have strengthened the signposting in Section 5 so that these diagnostic findings and their connection to the appendix analyses are immediately clear in the main text.

AuthorsOpenReview ↗

Response to Reviewer Nvxc

W4: Controlled ablation of ASR versus native audio

We conducted a controlled within-model modality ablation in Appendix D.4.4 (Table 17), evaluating three models under video-only (V), video with ASR transcripts (V+ASR), and native audiovisual input (V,A) while holding the backbone, request, and decision schedule fixed:

  • Auditory-triggered targets: Native omni input (V,A) outperforms both V and V+ASR for Gemini-3.5-Flash and Qwen3-Omni. Specifically, Gemini-3.5-Flash improves from 6/78 (V) and 6/78 (V+ASR) to 13/78 (V,A), while Qwen3-Omni improves from 4/78 (V) and 8/78 (V+ASR) to 11/78 (V,A) (representing a +8.97 percentage point gain over V for both models).
  • Overall modality ablation subset: Native V,A also yields more total hits than V+ASR across all three backbones: Gemini-3.5-Flash (322 vs. 316), Qwen3-Omni (125 vs. 118), and Qwen2.5-Omni (50 vs. 42).

Thus, ASR does not provide an overall advantage over native audio within any of the three tested backbones. The gains vary by trigger type: Qwen2.5-Omni ties ASR on auditory targets, and both Qwen models obtain one fewer hit with native audio on audiovisual targets. These controls do not establish ASR as the cause of the cross-model differences in Table 3. We have highlighted this ablation explicitly in Section 4.2.

W5: Human validation and consistency of GPT-4o scoring

We have added a stratified human evaluation of 140 model-response comparisons across T2 to T8 (20 comparisons per task; T1 uses deterministic exact-match scoring on tool invocations) in Appendix B.3.1 (Table 6):

  1. Human–Judge Correlation: An independent human reviewer graded the same 140 samples using our scoring rubrics. GPT-4o achieves 89.29% agreement (125/140) with human judgments, corresponding to Cohen's κ=0.786\kappa = 0.786 and a binary correlation (Phi / Pearson rr) of 0.7860.786. The 15 disagreement cases comprise seven false acceptances and eight false rejections. GPT-4o itself is not among the evaluated baseline models.
  2. Judge Repeatability: Across three independent evaluation runs at temperature zero, GPT-4o produces 132/140 unanimous verdicts (94.29%), with a mean per-sample binary variance of 0.0190, confirming that the scoring protocol is highly stable and aligned with human judgment.

W6: Dataset availability

We have added an explicit Data Availability statement to Appendix A.3 and the Reproducibility Statement. All 1,500 interaction annotations, evaluation scripts, and the raw video files for all 208 self-recorded first-person assistance videos will be publicly hosted on Hugging Face. For the 920 online videos, we will provide their original source URLs.

AuthorsOpenReview ↗

Response to Reviewer Nvxc

Responses to Questions

Q1: Video accessibility

Please refer to our Response to W6. The 208 self-recorded assistance videos will be hosted directly as downloadable video files in our Hugging Face release. For the 920 online videos, we will provide their original source URLs for access through the source platforms.

Q2: Video source selection

Please refer to our Response to W2. Online videos were retrieved by scene category and manually filtered for continuous causal structure, clear audiovisual quality, and identifiable event triggers. Self-recorded videos were captured from a first-person perspective across real-world assistive scenarios (T8-A/B/C). This pipeline yields a balanced distribution across environments (37.1% indoor, 34.2% outdoor, and 28.5% digital-screen interactions; Figure 1 and Appendix A.3).

Q3: Role of AI in preliminary data validation

During dataset construction, Gemini-3.5-Flash and Gemini-3.1-Pro assisted preliminary checks of formatting and label consistency in text metadata, flagging missing trigger-modality labels and grammatical ambiguities in user instructions (Appendix A.4). To be specific, (1) AI models never finalized annotations or timestamps, (2) no videos or interactions were filtered based on whether a model answered them correctly, and (3) human reviewers independently inspected every video stream and verified the exact timing and reference content of all 1,500 interactions. We have updated the manuscript to clarify this workflow.

Q4: Correlation between human evaluation and GPT-4o

Please refer to our Response to W5. On our stratified 140-sample benchmark audit across T2 to T8, GPT-4o verdicts show strong correlation with independent human annotations (r=0.786r = 0.786, Cohen's κ=0.786\kappa = 0.786, and 89.29% agreement). We have included the task-level breakdown in Appendix B.3.1 (Table 6).

Q5: ASR model details

We used Qwen3-ASR-1.7B to generate cached transcripts shared across all V+ASR model configurations, ensuring a fair comparison. For explicit-request tasks (T1 to T7), word-level timestamps were aligned using Qwen3-ForcedAligner-0.6B; for assistance tasks (T8), transcripts were extracted from completed 1-second causal audio chunks. We have added these exact model identifiers to Appendix C.1.

Reviewer_xx5BOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the OmniInteract-Scope benchmark with a mix of proactive assistance tasks: Event-Timed Tool Invocation, Event Notification, Risk & Error Interjection, Cumulative-State Tracking, Temporal Conditions, Nonverbal & Workflow Cues, Continuous Responses, Query-Free Reminders, Audiovisual Q&A, Multi-Step Guidance. This benchmarks tests the ability of models to know "when and what to say" in an omni modal setting where both visual input and auditory input is in play. The paper evaluates state of the art models such as Gemini-3.5-Flash on the proposed benchmark.

Critical strengths
  • The proposed benchmark OmniInteract-Scope is somewhat novel. It evaluates models on the ability to invoke the right tool at the right time, i.e., the "T1 Proactive Tool Invocation" task.

  • Th proposed OmniInteract-Scope benchmark does also include a novel mix of proactive assistance tasks: Event-Timed Tool Invocation, Event Notification, Risk & Error Interjection, Cumulative-State Tracking, Temporal Conditions, Nonverbal & Workflow Cues, Continuous Responses, Query-Free Reminders, Audiovisual Q&A, Multi-Step Guidance.

  • The OmniInteract-Scope benchmark is well presented.

Critical weaknesses
  • The proposed benchmark overlaps significantly with existing benchmarks in this area, especially with: "Can Vision-Language Models Answer Face to Face Questions in the Real-World?, ICLR 2026" and "Don't Pause: Every prediction matters in a streaming video, NeurIPS 2026". These benchmarks require the model to decide "When and What to say". These are the skills as those tested by the benchmark in this paper. Importantly, QICD [ICLR 2026] is also omni model -- it requires models to integrate visual and auditory cues for answering quetions.

  • The size of the benchmark is quite small. There are only 1500 questions across 8 tasks. So, the lack of data is very severe when we look at each task in isolation.

  • The benchmark evaluation is a bit limited. It does not consider state of art open source multi-modal LLMs such as Qwen3.5-9B or Qwen3.5-27B.

  • The evaluation of streaming models such as LiveCC is not clear. LiveCC is largely a streaming captioning model. How are the captions converted to decisions on the proposed OmniInteract-Scope dataset.

  • The paper could also include additional streaming models such as Videollm-online and ProAssist ("Proactive Assistant Dialogue Generation from Streaming Egocentric Videos", EMNLP 2025).

  • The paper should include more qualitative examples of results from the compared models such as Gemini-3.5-Flash and Qwen3-Omni.

Questions for authors
  • The paper should justify the benchmark size. Is the size sufficient for accurate evaluation not dominated by noise?
  • The paper should discuss prior work with more rigor.
  • The paper should also justify the baseline choices in more detail.
5 comments
AuthorsOpenReview ↗

Response to Reviewer xx5B

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript and providing constructive suggestions. We have revised the manuscript and Appendix accordingly. Below are our detailed point-by-point responses.

Responses to Weaknesses

W1: Distinctions from QIVD (ICLR 2026) and SPOT-Bench ("Don't Pause!")

We agree that deciding when and what to say in streaming video is a shared foundational goal, and we have updated Section 2 and Table 1 to explicitly discuss and credit both QIVD [1] and SPOT-Bench [2]. At the same time, our benchmark introduces three key dimensions that complement and extend these concurrent efforts:

  1. From Situated QA to Event-Timed Tool Invocation and Multi-Target Assistance: QIVD [1] centers on single-target, question-triggered situated QA. Our benchmark expands this scope to include event-timed external tool invocation (T1), continuous responses (T7) and multi-step guidance (T8-C) where a single request requires multiple timed interventions across a stream, and query-free impairment assistance (T8-A) driven by standing user needs rather than explicit questions.
  2. Native Audiovisual Grounding and Hearing Assistance vs. Visual-Only Streams: While SPOT-Bench [2] evaluates proactive intervention and sequential guidance, it operates on visual-only inputs. Our benchmark integrates native audiovisual cues across all tasks, specifically evaluating pure auditory and audiovisual event triggers as well as real-world hearing-impairment assistance (monitoring environmental alerts and speech for deaf or hard-of-hearing users).
  3. Counterfactual Diagnostics for Timing vs. Priors: Beyond aggregate accuracy, we provide matched counterfactual controls in Section 5 and Appendix D (no-scene ablations, event-shifted/absent counterfactuals, response-history conditioning, budget KK-sweeps, and autonomous vs. scheduled policies) that disentangle whether models respond due to genuine visual-auditory grounding, text-request priors, or high response frequency.

W2: Benchmark scale and evaluation units

We have clarified the video–interaction–target hierarchy in Section 3 and Appendix A.1 to avoid any ambiguity about scale:

  • The reported 1,500 entries are streaming interaction episodes constructed over 1,128 videos, comprising 1,200 explicit-request interactions in T1–T7 (with ~210 interactions in each of T1–T4) and 300 assistance interactions in T8.
  • Because continuous responses (T7) and multi-step guidance (T8-C) require models to respond at multiple distinct moments under a single request, these 1,500 interactions encompass 1,814 timestamped reference targets (1,453 in T1–T7 and 361 in T8; for instance, 209 interactions in T7 contain 462 individual evaluation targets).
  • Furthermore, unlike static QA, causal streaming evaluation requires dense second-by-second decision steps across the video stream, testing both when to respond and when to remain silent.

In terms of video collection, our benchmark (1,128 videos) exceeds established streaming benchmarks such as OVO-Bench (644 videos) [5] and StreamingBench (900 videos) [6], and is on par with OmniMMI (1,121 videos) [7].

W3: Evaluation of Qwen3.5-9B and four additional recent baselines

Following your suggestion, we have expanded the evaluation with five recent open-source and proprietary baselines, including Qwen3.5-9B, evaluated under the same causal streaming protocol and scorer across all 1,083 main-set interactions (1,336 targets). Joint content-and-timing accuracy (%) is reported below:

ModelT1T2T3T4T6T7Avg.
Qwen3.5-9B23.923.350.960.952.704.556.07
Muse Spark 1.131.5833.0113.4010.4827.0314.5021.67
Seed2.0 Lite30.1428.2313.406.1924.3212.3419.10
MiniMax-M329.1922.979.095.2424.3211.2617.01
JoyAI-VL-Interaction27.750.480.000.480.000.434.86

Note: Avg. denotes the unweighted mean over the six scored tasks; T5 remains a separate temporal-condition diagnostic.

These results reinforce our core findings: Qwen3.5-9B (6.07%) improves over earlier open-source counterparts under the same V+ASR interface (such as Qwen3-VL-8B at 5.0% and InternVL3.5-8B at 5.8%), while strong proprietary systems like Muse Spark 1.1 (21.67%) and Seed2.0 Lite (19.10%) perform competitively with Gemini-3.5-Flash (19.96%), yet still struggle substantially on risk/error interjection (T3), cumulative-state tracking (T4), and continuous responses (T7).

AuthorsOpenReview ↗

Response to Reviewer xx5B

W4: Evaluation protocol for LiveCC and streaming models

We have clarified the streaming interface in Section 4.1 and Appendix C.2. Crucially, we do not apply any post-hoc caption-to-decision converter, as doing so would conflate the evaluated model's intrinsic triggering policy with an external heuristic. Instead, LiveCC is prompted directly with the user request (and the candidate tool schemas in T1) as its system context and processes the video causally at 1 fps:

  • Triggering decision: At each step, if LiveCC emits its native silence token, it is recorded as withholding output. Any non-silent text generation is recorded as an active response (consecutive chunks generated within a 2-second window are merged into a single utterance timestamped at the onset of the first chunk).
  • Scoring: The emitted utterance is evaluated directly against the target time window and reference content/tool call.

This setup measures whether a streaming commentary model can adapt its native speaking policy to interactive instructions. Notably, our diagnostic in T1 reveals that LiveCC produces 81 timely responses within the ground-truth windows (showing sensitivity to visual state changes), but achieves 0% tool-invocation accuracy because it defaults to narrating the scene rather than formatting the required tool call.

W5: Relationship to VideoLLM-online and evaluation of ProAssist

We have added a detailed discussion of both VideoLLM-online [3] and ProAssist [4] in Section 2 and Appendix C.1 to contextualize our streaming baseline selection, and completed a ProAssist evaluation in Appendix D.5:

  1. Evolution from VideoLLM-online to LiveCC: LiveCC (CVPR 2025), which is evaluated in our benchmark, was developed by the same core research team as VideoLLM-online [3] as its direct next-generation successor, upgrading the frame-level streaming architecture with denser token compression and large-scale real-time commentary pretraining. Alongside MMDuet2, MOSS-VL, AURA, and MiniCPM-o 4.5, our five original streaming baselines represent the latest open-source evolution of stateful visual streaming and full-duplex audiovisual interaction.
  2. ProAssist Evaluation: We evaluated the released ProAssist-Model-L4096-I10 checkpoint under the same causal observation schedule and scorer, using visual input and request text without scene audio or additional task recipes. Across all 1,083 main-set interactions (1,336 targets), it achieves 0.08% Avg. (0.48% on T1, zero on T2/T3/T4/T6/T7). The separate T5 diagnostic yields 0/115. On all 300 T8 interactions (361 targets), timely target recall is 0% for T8-A, T8-B, and T8-C, with 0/26 Complete T8-C. Full settings and results are in Appendix D.5.

W6: Additional paired qualitative examples

We have added six new paired case studies in Appendix E.6 (Figures 12–17), beyond the existing examples. Each compares Gemini-3.5-Flash and Qwen3-Omni on the same interaction, with source frames, requests or standing instructions, reference windows, and timestamped model responses:

  1. Event timing (T2/T3): Gemini-3.5-Flash correctly times a nail-curing reminder and a hot-oil warning, while Qwen3-Omni acknowledges the request prematurely or remains silent. In the wasp-nest case, Qwen3-Omni gives a timely warning, whereas Gemini-3.5-Flash's first response is an early tool call, followed by relevant safety advice.
  2. Content and initiative (T7/T8-A): The chalk example contrasts narration of the current action with a process summary. The exposed-knife example distinguishes an actionable safety warning from a scene description that mentions the knife but omits the warning.
  3. Sequential coverage (T8-C): In bookstore navigation, Gemini-3.5-Flash covers all three reference steps, while Qwen3-Omni first responds after all three windows have elapsed.
AuthorsOpenReview ↗

Response to Reviewer xx5B

Responses to Questions

Q1: Statistical reliability and noise control

Please also refer to our Response to W2 regarding our 1,128 videos, 1,500 interactions, and 1,814 annotated targets. We assess evaluation reliability through model-tier comparisons and the scoring repeatability analysis in Appendix B:

  1. Substantial Performance Gaps: The main comparison covers 1,336 targets across 1,083 interactions, with a 13.39-percentage-point gap between Gemini-3.5-Flash (19.96%) and Qwen3-Omni (6.57%) in six-task Avg. T8 is assessed separately on 361 assistance targets.
  2. Minimal Scoring Variance: T1 is scored deterministically via exact match on tool invocations (zero judge variance). For T2 to T8, our temperature-zero rubric evaluation achieves 94.3% unanimous repeatability (132/140) across three repeated runs (mean per-sample variance of 0.0190) and 89.29% agreement (125/140, Cohen's κ\kappa = 0.786) with human annotators on a 140-sample stratified audit.

Q2: Rigorous discussion of prior work

Please refer to our Responses to W1 and W5. We have revised Section 2 and Table 1 to thoroughly compare our benchmark against concurrent situated QA and streaming benchmarks (including QIVD [1] and SPOT-Bench [2]) as well as proactive streaming assistants (VideoLLM-online [3] and ProAssist [4]), explicitly detailing how our work builds upon them while extending evaluation to native audio-visual triggers, tool invocation, hearing/vision assistance, and counterfactual policy diagnostics.

Q3: Justification of the 18 evaluated baselines

Please refer to our Responses to W3, W4, and W5. Building on Section 4.1 and Appendix C.1, we organize our 18 evaluated baselines (12 original + 5 newly added in W3 + ProAssist) into three complementary paradigms to ensure comprehensive coverage:

  1. Frontier Native-Omni & Proprietary Systems (Gemini-3.5-Flash, Gemini-3.1-Pro, Qwen3.5-Omni-Plus, Muse Spark 1.1, Seed2.0 Lite, MiniMax-M3): Establish strong reference performance on joint temporal-semantic reasoning and audiovisual integration.
  2. Open-Source General MLLMs & Omni-LLMs (Qwen3-Omni, Qwen2.5-Omni, Qwen3.5-9B, Qwen3-VL, InternVL3.5, JoyAI-VL-Interaction): Evaluated under unified causal prompting with V, V+ASR, or native Omni inputs to compare general-purpose visual-language and omni backbones.
  3. Specialized Streaming & Duplex Models (LiveCC, MMDuet2, MOSS-VL, AURA, MiniCPM-o 4.5, ProAssist): Evaluate architectures designed for frame-level silence/speaking decisions, rolling-window memory, or full-duplex turn-taking.

References

[1] Pourreza et al. Can Vision-Language Models Answer Face to Face Questions in the Real-World? ICLR 2026.
[2] Chatterjee et al. Don't Pause! Every prediction matters in a streaming video. NeurIPS 2026.
[3] Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024.
[4] Zhang et al. Proactive Assistant Dialogue Generation from Streaming Egocentric Videos. EMNLP 2025.
[5] Niu et al. OVO-Bench: How Far Is Your Video-LLMs from Real-World Online Video Understanding? CVPR 2025.
[6] Lin et al. StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding. ICASSP 2026.
[7] Wang et al. OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts. CVPR 2025.

Reviewer_3Xw6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Omni-InteractScope, a benchmark of 1,500 annotated interactions for evaluating proactive assistance in audiovisual streams. It evaluates 12 model configurations on response content, timing, and coverage across eight task groups.

Critical strengths
  1. The paper constructs a human-annotated benchmark for proactive multimodal interaction.
  2. The authors evaluate multiple models and provide useful diagnostic experiments.
Critical weaknesses
  1. The tool-use evaluation is too limited. It covers only three search tools and checks tool names and invocation timing, without evaluating arguments or actual execution.
  2. Tool-invocation scores do not reliably demonstrate the use of scene evidence. The paper’s own controls show that removing scene input does not reduce performance, raising concerns about what this task actually measures.
  3. The assistance metric ignores unnecessary interventions. A model can produce many incorrect or redundant responses without reducing its recall score, limiting the metric’s usefulness for evaluating proactive assistance.
  4. The automatic scoring lacks sufficient validation. Human checks of the annotations do not establish whether the LLM judge reliably evaluates model responses.
Questions for authors
  1. The benchmark includes only image search, web search, and audio search. How were these tools selected, and how well do they represent the agentic capabilities discussed in the paper? Can the evaluation include more diverse tools and assess whether tool calls are correctly executed?
  2. In Section 5.1, Gemini-3.5-Flash scores 32.1% without scene input versus 28.7% with it. How can the authors ensure that success on T1 requires observing the target event? For example, could they hold the request fixed while removing the event or changing when it occurs?
  3. Table 10 reports the same target recall for Gemini-3.5-Flash while producing approximately 58 and 24 responses per minute under different history settings. Can the authors evaluate whether the additional responses are useful, incorrect, or redundant, and compare models under matched response budgets?
  4. Could the authors report agreement between human reviewers and GPT-4o on a sample of model responses? Since the judge sees textual references rather than the original audiovisual input, it is important to check whether it rejects unsupported details and accepts valid alternative answers.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Xw6

We thank the reviewer for taking the time and effort to review our manuscript. Below we address each weakness and question in detail.

Responses to Weaknesses

W1: Scope of tool evaluation

T1 (Event-timed tool invocation) is designed to isolate two foundational prerequisites of proactive tool use in continuous streams: deciding when to act based on unfolding events and selecting which retrieval channel to invoke. The three tools map directly to the visual, textual, and acoustic modalities of omnimodal streams. Under our strict first-response scoring rule (Section 4.1, Appendix B.4), a premature call fails the target even if a later call is correct. Even without grading argument syntax or live API execution, all 12 models in the timing decomposition produce an on-time first response on fewer than 40% of targets (Appendix D.1). Across the expanded model comparison, joint tool-and-timing accuracy peaks at only 31.6% (Table 2). This shows that timely invocation itself remains a substantial challenge for current Omni-LLMs, even before argument correctness and downstream execution are evaluated.

W2: Scene grounding and diagnostic controls

Showing why raw invocation accuracy alone does not establish scene grounding is one of our core diagnostic contributions (Section 5.1, Figure 3a). In our paired ablation on the same 209 interactions, Gemini-3.5-Flash scores 28.71% with scene input and 32.06% without it, while request text alone suffices to recover 202/209 tool labels (Appendix D.1.1). Rather than treating raw T1 accuracy as standalone proof of perceptual grounding, we introduce these fixed-delay and no-scene controls (Appendices D.1.1 and D.1.2, respectively) specifically to show that tool-and-time matches can arise from request and timing priors without observing the target event. We therefore interpret T1 scores alongside these diagnostic controls. Our new event-shifted and event-absent counterfactuals provide direct evidence of event-sensitive timing under fixed requests, as detailed in our Response to Q2 and Appendix D.1.3.

W3: Handling unnecessary interventions and output volume

In continuous real-world streams, reference annotations mark essential target reminders rather than exhaustively cataloging every valid helpful remark across a multi-minute video (Appendix B.5). Automatically penalizing all unmatched outputs as false positives would unfairly punish valid, unannotated assistance. To account for over-generation without mislabeling valid outputs, we pair target recall with output volume (outputs per minute) and evaluate explicit response-control policies (Section 5.2, Appendix D.2.2). For example, supplying recent response history cuts Gemini-3.5-Flash's output volume by 58.1% (from 7,833 to 3,283 outputs) while preserving aggregate target recall (22/122). We also compare recorded trajectories under common per-episode response caps and temporal spacing filters in Appendix D.2.3 (see our Response to Q3).

W4: Validation of LLM-as-Judge scoring

To validate our GPT-4o content judge (which follows established reference-based judging protocols in streaming benchmarks [1, 2]), we conducted a stratified human audit of 140 model-response comparisons across T2 to T8 (20 instances per task; T1 is excluded as it uses deterministic exact-match scoring on tool sets):

  1. Human-Judge Agreement: GPT-4o agrees with human judgments on 125/140 comparisons (89.29% agreement, Cohen's κ\kappa = 0.786). Task-level agreement breakdowns are detailed in Appendix B.3.1.
  2. Judge Repeatability: Across three separate evaluation rounds at temperature zero, GPT-4o yields 132/140 unanimous verdicts (94.3%), with a mean per-sample unbiased binary variance of 0.0190.

We have added these validation results to Appendix B.3.1.

AuthorsOpenReview ↗

Response to Reviewer 3Xw6

Responses to Questions

Q1: Tool selection and execution

As discussed in our response to W1, the three search tools were selected to represent cross-modal information retrieval across visual entities, general web knowledge, and audio events. They test a core proactive behavior: recognizing when external information is required by a user's standing request and routing that trigger to the appropriate modality. Extending the evaluation to broader action spaces and live tool execution would additionally require ground-truth arguments and validation of tool outputs or resulting environment states. Because Omni-InteractScope evaluates causal observation over pre-recorded real-world videos, we focus T1 on event-timed invocation decisions and view closed-loop execution as a complementary direction explored in interactive environments.

Q2: Event counterfactuals

Following your suggestion, we added a counterfactual control on 30 T1 source videos in Appendix D.1.3, covering 2 unchanged requests and 28 explicitly conditional rewrites. Within each case, we hold the request and a 5s audiovisual prefix fixed across 25s clips under three conditions: an early event, the same event delayed by 5s, and no event. Note that the 5s delay exceeds twice our ±2s\pm 2\text{s} evaluation tolerance.

  • Conditional Rewrites: In 24 of 28 cases, Gemini-3.5-Flash invokes a tool within both the early and delayed event windows, with 20 of 28 also matching the annotated tool. When the event is removed, 23 of 28 correctly remain silent. Overall, 18 of 28 satisfy all three conditions jointly, and this result holds even under a stricter [onset, onset + 2s] window.
  • Unchanged Requests: Both cases shift their first tool call by 5s when the event is delayed, demonstrating that the model tracks the visual event onset. Neither satisfies all three conditions jointly: one calls a tool when the event is absent, while the other correctly abstains without the event but selects web_search instead of image_search.

The first tool-call time shifts by exactly five seconds in both unchanged requests and 26 of 28 conditional rewrites, supporting event-sensitive timing under these controlled conditions.

Q3: Extra responses and matched budgets

In the regenerated history controls, response history reduces Gemini-3.5-Flash's output volume by 58.1% (from 7,833 to 3,283) while preserving aggregate target recall (22/122). The matched targets differ: 10 are shared, with 12 gained and 12 lost. Output counts alone do not establish whether the additional responses are useful, incorrect, or redundant.

On the separate main-evaluation trajectories, a five-second minimum-spacing filter reduces Gemini-3.5-Flash's outputs from 7,813 to 1,642 while retaining 11 of 27 target hits, and Qwen3-Omni's outputs from 8,213 to 1,666 while retaining 2 of 3 hits. Thus, spacing reduces output volume substantially but also lowers target coverage.

To compare models under matched response budgets, we applied uniform post-hoc caps retaining at most the first K∈{1,3,5,10}K \in \{1, 3, 5, 10\} responses per episode to the original 12 models' T8-A trajectories. Applying K=5K=5 to the regenerated Gemini-3.5-Flash history controls retains 610 outputs per condition, covering 10/122 targets without history and 8/122 with history. We have added the full 12-model cap comparison and the spacing analyses to Appendix D.2.3.

Q4: Human-judge agreement

Please refer to our Response to W4, where we report a 140-sample human audit showing 89.29% overall agreement between human reviewers and the GPT-4o judge. We have also added the granular subgroup breakdown results to Appendix B.3.1.

References

[1] Niu et al. OVO-Bench: How Far Is Your Video-LLMs from Real-World Online Video Understanding? CVPR, 2025.
[2] Wang et al. OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts. CVPR, 2025.

AuthorsOpenReview ↗

Reminder to Reply

Dear Reviewer 3Xw6,

As the discussion period is nearing its end with less than one day, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. We are eager to address any remaining issues to improve our work.

Thanks again for your time and effort in reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear PC, SAC, AC, and Reviewers,

We sincerely appreciate the great effort and time you have spent reviewing each submission. Below, we summarize the core contributions of our work and the revisions made in response to the reviews and subsequent discussion.


Recognition from Reviewers

  • Reviewer 3Xw6: Recognized our "human-annotated benchmark for proactive multimodal interaction", multi-model evaluation, and "useful diagnostic experiments."
  • Reviewer xx5B: Noted that the benchmark is "well presented", tests "the ability to invoke the right tool at the right time", and offers "a novel mix of proactive assistance tasks."
  • Reviewer Nvxc: Commended the "human annotations", "diverse range of tasks and scene distributions", and broad evaluation across open/closed-source and streaming models.

Main Contributions of Our Work

  1. A novel benchmark for proactive and agentic interaction: Moving beyond reactive Q&A, our Omni-InteractScope curates 1,500 human-annotated episodes across 1,128 videos (1,814 temporal targets in eight task groups) to jointly test event-timed tool invocation, query-free intervention, and sustained multi-step guidance, going beyond the scope of prior benchmarks.
  2. Rigorous and controlled behavioral diagnostics: Using event-timing counterfactuals, modality and scene ablations, and response-budget controls, we disentangle genuine stream grounding from language priors and over-triggering heuristics.
  3. Broad evaluation of 18 VLMs and Omni-LLMs: Benchmarking proprietary, open-source, and specialized streaming models under a causal protocol exposes key bottlenecks: joint tool-and-timing accuracy peaks at 31.6%, while native audio processing consistently outperforms cascaded ASR.

Summary of Revisions and Responses to Reviewers

  • Reviewer 3Xw6

    • Tool scope & event grounding (§5.1, App. D.1): Clarified that T1 tests tool selection and timing. New controls on 30 videos hold requests fixed across early, delayed, and absent events to separate event-grounded timing from request/timing priors.
    • Unnecessary interventions (§5.2, App. D.2): Added shared response caps and temporal-spacing controls. In paired history controls, Gemini-3.5-Flash produces 58.1% fewer outputs while fully preserving reminder recall.
    • Scoring reliability (App. B.3.1): Reported 89.29% human–GPT-4o agreement (140 comparisons) and 94.29% 3-run consistency, alongside Claude Opus 5 cross-judge validation (see our response to Reviewer Nvxc).
  • Reviewer xx5B

    • Positioning & structure (§2, Table 1, App. A): Expanded comparisons with QIVD and SPOT-Bench (event-timed tools, native audio triggers, multi-target episodes) and clarified the hierarchy of videos, episodes, and targets.
    • Adding 6 new baselines (Table 2, App. E): Added Qwen3.5-9B, Muse Spark 1.1, Seed2.0 Lite, MiniMax-M3, JoyAI-VL-Interaction, and ProAssist, expanding our evaluation from 12 to 18 models under the same causal protocol.
    • Streaming protocols & case studies (App. C, E.6): Documented how native silence/speaking decisions are evaluated relative to VideoLLM-online and ProAssist, and added six visual case studies with timestamped outputs.
  • Reviewer Nvxc

    • Annotation pipeline (App. A.3–A.4): Detailed video sourcing, our 40-person annotation team, training, and cross-checking, clarifying that all reference content and timestamps are human-authored (LLMs assisted only with initial metadata screening).
    • Audio modality & diagnostics (§5, App. D): Strengthened analyses of response policies, sequence coverage, and timing tolerances. Controlled comparisons (video, video+ASR, native audiovisual) show native audio outperforms ASR across all three tested backbones.
    • Cross-judge validation & audit scope (App. B.3.1, Table 7): Explicitly stated that human validation used a single annotator and noted the lack of inter-annotator agreement as a limitation. Added Claude Opus 5 rescoring of 11 models, showing 89.99% agreement (κκ = 0.793, rr = 0.797), Pearson rr = 0.999 on Avg. scores, and identical rankings (Spearman ρρ = 1.000).
    • Data access (App. A.3, Reproducibility Statement): Confirmed public Hugging Face release of all annotations, evaluation scripts, 208 self-recorded videos, and source URLs for the 920 online videos.

These experiments, clarifications, and validation results are incorporated into the revised manuscript. We believe they address the main concerns raised in the reviews and strengthen our benchmark. We respectfully request favorable consideration of our work.

Thank you for your time and consideration.

Sincerely,
Authors of Submission 25856

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.