- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Omni-InteractScope, a benchmark aimed to measure how models can intervene on the correct timing based on stream evidence and user need. The dataset was created using online videos and human annotations and presents a diversity of tasks and scene distributions, from tool invocation to nonverbal and impairment assistance. 12 models are evaluated using this benchmark. The paper presents the following contributions: benchmark with multiple evaluation subtasks and evaluation on 12 multimodal models.
Critical strengths
- The dataset is created using human annotations.
- The dataset gathers a diverse range of tasks and scene distributions
- The evaluation is done with a broad range of models, including open/closed source models, different families and streaming models
Critical weaknesses
- The paper is not easy to follow and lacks clarity. This includes no reference to the number of annotators, how were the annotators selected. If being human annotated is part of the contributions, then this needs to be as much clear as possible.
- The dataset construction and justification on the visual data used is vague or non existent.
- The analysis is majorly focused on the main results and there is just a brief analysis beyond that.
- There is no ablation study to understand if non-omni models perform better than omni because of the ASR model used.
- Most tasks (T2-T8) are evaluated using a closed source model (gpt-4o) without cross human or model correlation.
- There is no claim regarding the availability of the dataset which makes the major contribution weaker.
Questions for authors
- Will the videos be part of the data? If not are they accessible to everyone?
- What was the selection method for the sources of the videos?
- How was AI used for data validation (mentioned in the AI Statement) before final human verification?
- Can you do the evaluation with humans or other model and measure the correlation with gpt-4o? It is fundamental to understand if the results presented are biased to the model or if the results generalize with other judges.
- Which ASR was used?
Suggestions:
- Task numbering should follow task clustering so this makes it easier to follow and understand by looking at the text and figure in the same order.
- Line 263 misses a bold on T1.
- Referencing a task of the dataset as “Our T1” (L.149) confounds more then helps.
- Digging more into analysis such as section 5 would make the paper much stronger.
- If a generic model + ASR is better then this should be more relevant on the results and analysis.
8 comments
Response to Reviewer Nvxc
W4: Controlled ablation of ASR versus native audio
We conducted a controlled within-model modality ablation in Appendix D.4.4 (Table 17), evaluating three models under video-only (V), video with ASR transcripts (V+ASR), and native audiovisual input (V,A) while holding the backbone, request, and decision schedule fixed:
- Auditory-triggered targets: Native omni input (V,A) outperforms both V and V+ASR for Gemini-3.5-Flash and Qwen3-Omni. Specifically, Gemini-3.5-Flash improves from 6/78 (V) and 6/78 (V+ASR) to 13/78 (V,A), while Qwen3-Omni improves from 4/78 (V) and 8/78 (V+ASR) to 11/78 (V,A) (representing a +8.97 percentage point gain over V for both models).
- Overall modality ablation subset: Native V,A also yields more total hits than V+ASR across all three backbones: Gemini-3.5-Flash (322 vs. 316), Qwen3-Omni (125 vs. 118), and Qwen2.5-Omni (50 vs. 42).
Thus, ASR does not provide an overall advantage over native audio within any of the three tested backbones. The gains vary by trigger type: Qwen2.5-Omni ties ASR on auditory targets, and both Qwen models obtain one fewer hit with native audio on audiovisual targets. These controls do not establish ASR as the cause of the cross-model differences in Table 3. We have highlighted this ablation explicitly in Section 4.2.
W5: Human validation and consistency of GPT-4o scoring
We have added a stratified human evaluation of 140 model-response comparisons across T2 to T8 (20 comparisons per task; T1 uses deterministic exact-match scoring on tool invocations) in Appendix B.3.1 (Table 6):
- Human–Judge Correlation: An independent human reviewer graded the same 140 samples using our scoring rubrics. GPT-4o achieves 89.29% agreement (125/140) with human judgments, corresponding to Cohen's and a binary correlation (Phi / Pearson ) of . The 15 disagreement cases comprise seven false acceptances and eight false rejections. GPT-4o itself is not among the evaluated baseline models.
- Judge Repeatability: Across three independent evaluation runs at temperature zero, GPT-4o produces 132/140 unanimous verdicts (94.29%), with a mean per-sample binary variance of 0.0190, confirming that the scoring protocol is highly stable and aligned with human judgment.
W6: Dataset availability
We have added an explicit Data Availability statement to Appendix A.3 and the Reproducibility Statement. All 1,500 interaction annotations, evaluation scripts, and the raw video files for all 208 self-recorded first-person assistance videos will be publicly hosted on Hugging Face. For the 920 online videos, we will provide their original source URLs.
Response to Reviewer Nvxc
Responses to Questions
Q1: Video accessibility
Please refer to our Response to W6. The 208 self-recorded assistance videos will be hosted directly as downloadable video files in our Hugging Face release. For the 920 online videos, we will provide their original source URLs for access through the source platforms.
Q2: Video source selection
Please refer to our Response to W2. Online videos were retrieved by scene category and manually filtered for continuous causal structure, clear audiovisual quality, and identifiable event triggers. Self-recorded videos were captured from a first-person perspective across real-world assistive scenarios (T8-A/B/C). This pipeline yields a balanced distribution across environments (37.1% indoor, 34.2% outdoor, and 28.5% digital-screen interactions; Figure 1 and Appendix A.3).
Q3: Role of AI in preliminary data validation
During dataset construction, Gemini-3.5-Flash and Gemini-3.1-Pro assisted preliminary checks of formatting and label consistency in text metadata, flagging missing trigger-modality labels and grammatical ambiguities in user instructions (Appendix A.4). To be specific, (1) AI models never finalized annotations or timestamps, (2) no videos or interactions were filtered based on whether a model answered them correctly, and (3) human reviewers independently inspected every video stream and verified the exact timing and reference content of all 1,500 interactions. We have updated the manuscript to clarify this workflow.
Q4: Correlation between human evaluation and GPT-4o
Please refer to our Response to W5. On our stratified 140-sample benchmark audit across T2 to T8, GPT-4o verdicts show strong correlation with independent human annotations (, Cohen's , and 89.29% agreement). We have included the task-level breakdown in Appendix B.3.1 (Table 6).
Q5: ASR model details
We used Qwen3-ASR-1.7B to generate cached transcripts shared across all V+ASR model configurations, ensuring a fair comparison. For explicit-request tasks (T1 to T7), word-level timestamps were aligned using Qwen3-ForcedAligner-0.6B; for assistance tasks (T8), transcripts were extracted from completed 1-second causal audio chunks. We have added these exact model identifiers to Appendix C.1.
Response to Reviewer Nvxc
We appreciate your time and effort in reviewing our manuscript. Below are our detailed point-by-point responses.
Responses to Weaknesses
W1: Annotator details and presentation clarity
We have updated Section 3.3 and Appendix A.3 to explicitly document our annotation team and quality-control workflow, and we have reorganized Section 3.2 to follow a strict T1 to T8 sequence (matching Figure 2 and Table 4) for better readability:
W2: Dataset construction and visual sources
We have expanded Appendix A.3, complementing Section 3.3, to detail the video acquisition, screening, and annotation pipeline across our 1,128 continuous videos:
W3: Depth of diagnostic analysis
Beyond the main benchmark rankings (Section 4.2), Section 5 and Appendix D (Tables 8 to 17) provide extensive controlled diagnostics into why current Omni-LLMs fail:
We have strengthened the signposting in Section 5 so that these diagnostic findings and their connection to the appendix analyses are immediately clear in the main text.