- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new benchmark to evaluate whether streaming omnimodal models can continuously process audio-visual inputs, maintain relevant context, and provide proactive responses to user queries. It includes a large-scale training dataset with approximately 200K spoken user-query turns, along with 3,419 evaluation query turns across five task families and 15 subtasks.
The experimental results show that existing streaming omnimodal models still struggle on this benchmark, particularly when handling audio queries compared to text queries. For example, the content accuracy of Qwen3.8-Omni Flash Realtime drops from 64.5% with text queries to 42.8% with audio queries. The authors also fine-tune a Qwen2.5-Omni model using their proposed training dataset and demonstrate improvements in performance.
Critical strengths
- The paper constructs a large-scale training dataset and an evaluation benchmark for streaming audio-visual question answering.
- As the authors highlight in the related work, the proposed benchmark provides a more fine-grained evaluation by covering a broader range of subtasks compared to existing benchmarks. In addition, the authors aim to distribute query timestamps throughout the video, rather than placing queries primarily at the beginning.
- The paper is well-organized and easy to follow.
Critical weaknesses
I think there are several aspects of this paper that could be improved.
-
Lack of human evaluation for data quality. The authors use AI models and other automated methods for quality control. However, for a benchmark paper, I believe it is still necessary to include human evaluation on at least a subset of the data. For example, human annotators could review the generated questions and answers to verify their correctness. This would help assess whether the automatically generated data and the overall construction pipeline are trustworthy.
-
Limited distinction from existing benchmarks and lack of new insights. Several existing works, such as OmniInteract (Lu et al., 2026), ROMA (Tian et al., 2026), OmniAssistBench (Sun et al., 2026), and ProAssist (2025), have already explored similar streaming audio-visual interaction settings. I understand that the authors claim to provide a more fine-grained evaluation with a broader range of subtasks compared to existing benchmarks. However, I do not see clear new insights emerging from this new benchmark setting. It would be helpful to demonstrate what additional challenges or limitations of existing models can be revealed by the proposed benchmark but not by previous ones.
-
Lack of direct comparison with existing training datasets. A more controlled comparison would help demonstrate the value of the proposed training dataset. For example, the authors could fine-tune the same backbone model using NativeQuery-Train and another existing training dataset (e.g., ROMA's training dataset), respectively, and compare their performance under the same evaluation settings. This would help clarify whether the proposed dataset provides additional benefits over existing training datasets.
Overall, I think the paper would benefit from more experimental comparisons and deeper analysis to demonstrate the value of the newly constructed benchmark. Simply introducing a larger benchmark with more subtasks is not sufficient without showing what new insights or capabilities it enables.
My main concerns can be summarized from two perspectives:
-
Benchmark quality and reliability: Is the benchmark trustworthy? How reliable are the automatically generated questions, answers, and annotations?
-
Benchmark value and necessity: What additional value does this benchmark provide compared to existing works? Does it reveal new insights into the limitations of current streaming models, or enable new capabilities through its proposed training dataset?
Questions for authors
-
Benchmark Quality: Could the authors clarify whether they can provide human evaluation results? In particular:
- (a) Have human annotators independently reviewed a subset of the benchmark to verify the quality of the generated annotations?
- (b) Could the authors conduct a human evaluation on a subset of the QA tasks and report the results, particularly regarding answer correctness, evidence grounding, and response timing?
-
Novelty and Insights: Beyond the broader task coverage and more distributed query timestamps, what new insights does NativeQuery provide compared to existing benchmarks? Are there any specific model limitations or failure cases that NativeQuery reveals but existing benchmarks cannot adequately capture?
-
Training Data Effectiveness: Could the authors provide a more controlled comparison using the same backbone model fine-tuned on different training datasets under comparable training settings? This would help clarify whether the performance improvements come from the proposed dataset itself or simply from additional fine-tuning.
5 comments
Response to Reviewer 59qq (Part 2 / 3)
W2.2 & Q2. New Insights and Model Limitations or Failure Cases.
A4. NativeQuery reveals three important limitations that are not captured by existing evaluation settings.
First, explicit-query performance can hide failures in spoken-query understanding. We compare identical questions in text and audio forms. Existing models show substantial performance drops when queries are embedded in continuous audio-visual streams. NativeStream significantly reduces this gap. This demonstrates that understanding spoken requests within a stream remains a distinct challenge.
| Model | Text C (%) ↑ | Audio C (%) ↑ |
|---|---|---|
| Qwen3.8-Omni Flash R | 64.5 | 42.8 |
| NativeStream-7B | 66.8 | 63.2 |
Second, strong grounded reasoning does not guarantee reliable handling of ambiguity and changing user intent. Model performance varies substantially across these tasks. MiniCPM-o outperforms Qwen3.5-Omni on GrdRe but performs worse on DynMan. NativeStream achieves much larger improvements on ConAmb and DynMan than on GrdRe. These results show that grounded QA performance alone cannot reflect a model's ability to manage evolving user requests.
| Model | GrdRe J (%) ↑ | ConAmb J (%) ↑ | DynMan J (%) ↑ |
|---|---|---|---|
| Qwen3.5-Omni Plus R | 22.0 | 12.5 | 11.3 |
| MiniCPM-o-4.5-9B | 44.5 | 19.7 | 8.1 |
| NativeStream-7B | 45.2 | 36.7 | 25.3 |
Third, correct response timing does not guarantee reliable interaction. ROMA-7B achieves slightly higher timing accuracy than MiniCPM-o, but its timing precision and content accuracy are much lower. This shows that models may respond at the expected times but still produce incorrect or unnecessary responses.
| Model | T Acc. (%) ↑ | T Prec. (%) ↑ | C (%) ↑ | J (%) ↑ | JF1 (%) ↑ |
|---|---|---|---|---|---|
| ROMA-7B | 49.9 | 11.1 | 9.9 | 5.2 | 2.0 |
| MiniCPM-o-4.5-9B | 47.2 | 25.4 | 54.8 | 26.7 | 18.4 |
| NativeStream-7B | 57.8 | 30.1 | 63.2 | 36.2 | 24.7 |
W3.1 & Q3. Controlled Comparison with Existing Training Datasets.
A5. We thank the reviewer for this suggestion. We would like to clarify that ROMA's publicly released training data are incomplete. The public release contains only the proactive subset, while the Reactive QA subset is not available. Fine-tuning on the released data would therefore not reproduce ROMA's full training setup.
| Training Subset | Original ROMA | Public Release |
|---|---|---|
| Reactive QA | 540,538 | Not released |
| Event-Driven Alert | 27,207 | 27,207 |
| Real-Time Narration | 108,986 | 108,986 |
| Total | 676,731 | 136,193 |
Nevertheless, ROMA-7B and NativeStream-7B share the same Qwen2.5-Omni-7B backbone. As reported in Table 1 of our paper, we evaluate both models on NativeQuery-Bench using the same Streaming Mode setting. NativeStream-7B substantially outperforms ROMA-7B across all four metrics. This comparison demonstrates the stronger performance of our trained model on in-stream spoken-query interaction.
| Model | T ↑ | C ↑ | J ↑ | JF1 ↑ |
|---|---|---|---|---|
| ROMA-7B | 49.9 | 9.9 | 5.2 | 2.0 |
| NativeStream-7B | 57.8 | 63.2 | 36.2 | 24.7 |
To separate the contribution of NativeQuery-Train from the effect of additional fine-tuning, we also train a text-query control. It uses the same backbone, NativeQuery-Train samples, and training settings as NativeStream-7B. The only difference is that spoken queries are replaced with text, so both models receive the same amount of fine-tuning on the same data. We evaluate both models in Streaming Mode with text and spoken queries.
| Training Query | Test Query | T ↑ | C ↑ | J ↑ | JF1 ↑ |
|---|---|---|---|---|---|
| Text | Text | 63.7 | 69.6 | 44.0 | 29.8 |
| Text | Spoken | 34.2 | 41.6 | 14.6 | 9.2 |
| Spoken | Text | 59.1 | 66.8 | 39.2 | 26.3 |
| Spoken (NativeStream-7B) | Spoken | 57.8 | 63.2 | 36.2 | 24.7 |
The control performs well on text queries (44.0% J), but reaches only 14.6% J on spoken queries, compared with 36.2% for NativeStream-7B. This shows that the improvement comes from the in-stream spoken-query supervision in NativeQuery-Train, not from additional fine-tuning alone.
Response to Reviewer 59qq (Part 3 / 3)
W3.2 & Q3. Training Data Effectiveness vs. Additional Fine-Tuning.
A6. We further compare NativeStream-7B with the original Qwen2.5-Omni-7B to assess the effectiveness of NativeQuery-Train. Since the base model cannot autonomously decide when to respond, we evaluate both models in Oracle Mode. Both receive the same audio-visual inputs and follow the same evaluation protocol.
Fine-tuning on NativeQuery-Train substantially improves the backbone's ability to understand spoken requests embedded in continuous audio-visual streams. The fine-tuned model also exceeds the base model's text-query accuracy. Together with the ROMA comparison, these results support the effectiveness of NativeQuery-Train.
| Model | Query Type | C (%) ↑ |
|---|---|---|
| Qwen2.5-Omni-7B | Text | 62.0 |
| Qwen2.5-Omni-7B | Audio | 36.7 |
| NativeStream-7B | Audio | 63.9 |
Response to Reviewer 59qq (Part 1 / 3)
We sincerely thank the Reviewer 59qq for the thorough and constructive review. We appreciate the reviewer’s recognition that NativeQuery goes beyond existing benchmarks with a more fine-grained evaluation, covering a broader range of subtasks and distributing queries throughout the video. We are also grateful that the reviewer acknowledges its accompanying training dataset for streaming audio-visual question answering. We respond to the questions raised by the reviewer point by point below.
A1. Thank you for raising this concern. We would like to clarify that human validation is part of our NativeQuery-Bench construction pipeline. It takes place after MLLM-based verification. Ten English-proficient annotators participate in this process. Each sample is independently reviewed by two annotators. A third annotator resolves any disagreements.
The annotators review the audio-visual streams to verify answer correctness, evidence grounding, and query/response timing. They check whether the answers are supported by the source evidence. They also check whether the annotated timestamps are consistent with evidence availability and task requirements. Problematic annotations are corrected or discarded. We have clarified these details in Appendix B of the revised paper.
A2. We thank the reviewer for this suggestion. We report the verification results across all five tasks. Of the 5,000 initial candidates, 3,449 pass MLLM-based verification. After human validation, 2,735 samples are retained. This shows that human review provides an additional quality check beyond automated verification. We have included the detailed verification statistics in Appendix B of the revised paper.
We further assess the reliability of our automatic content evaluator, GPT-5.6-sol. We sample 300 timely model responses, with 60 from each task. Two annotators independently judge each response as correct or incorrect. A third annotator resolves any disagreements. The evaluator scores responses from 0 to 5. Scores of 3 or higher are considered correct. We compare its judgments with the final human labels. The evaluator achieves an agreement rate of 89.3% and a Cohen's κ of 0.75, with substantial agreement across all five tasks. These results show that the evaluator's judgments are largely consistent with human judgments.
A3. We thank the reviewer for this comment. NativeQuery differs from existing benchmarks not simply in task coverage, but in the interaction challenges it evaluates.
(1) Compared with OmniInteract, NativeQuery places greater emphasis on cross-modal reasoning and dynamic user intent. It verifies that both audio and visual evidence are necessary to answer each cross-modal question. It also includes 320 forbidden response events to evaluate whether models can suppress responses to outdated requests after users change their intent.
(2) Unlike ROMA, which uses synthesized query voices that differ from the original video speakers, NativeQuery uses voice matching with the original speakers. This reduces obvious changes in voice that models could otherwise use to detect queries.
(3) Compared with OmniAssistBench, which uses segmented videos and offline inference, NativeQuery preserves continuous audio-visual streams and evaluates models through native online inference.
(4) Unlike ProAssist, which provides user goals and follow-up requests as text, NativeQuery embeds spoken requests directly into the audio stream. Models must recognize these requests while processing the ongoing audio-visual input.