acceptodds
Under review as a conference paper at ICLR 2027

NativeQuery: Bringing Spoken Queries into Continuous Audio-Visual Streams

Abstract

Streaming omnimodal models can continuously perceive audio and visual inputs, but user queries are still typically provided separately as explicit inputs rather than arising within the stream itself. In real-world interactions, queries naturally occur as part of the continuous audio-visual stream, requiring models to recognize and interpret them while continuously processing the surrounding multimodal context. To address this challenge, we introduce NativeQuery, consisting of NativeQuery-Train, a large-scale training corpus, and NativeQuery-Bench, an evaluation benchmark for spoken-query interaction in continuous audio-visual streams. NativeQuery-Train contains 200K user-query turns, while NativeQuery-Bench contains 3,419 evaluation user-query turns across five task families and fifteen subtasks, covering grounded multimodal reasoning, conflict and ambiguity handling, persistent task execution, dynamic task management, and situated interaction reasoning. NativeQuery integrates queries into the original streams while preserving their native speakers and surrounding audio-visual context. We further introduce NativeStream, a streaming omnimodal model trained on NativeQuery-Train for such in-stream interaction. Experiments on NativeQuery-Bench reveal a substantial gap between explicit text queries and in-stream spoken queries, with Qwen 3.8 Omni Flash Realtime dropping from 64.5% to 42.8% in content accuracy. NativeStream substantially mitigates this problem, with NativeStream-7B achieving 36.2% joint accuracy on NativeQuery-Bench and 0.419 All-Global on OmniInteract, outperforming existing streaming omnimodal models.

Primary area:
datasets and benchmarks
Keywords:
omnimodal, streaming, spoken interaction

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_59qqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new benchmark to evaluate whether streaming omnimodal models can continuously process audio-visual inputs, maintain relevant context, and provide proactive responses to user queries. It includes a large-scale training dataset with approximately 200K spoken user-query turns, along with 3,419 evaluation query turns across five task families and 15 subtasks.

The experimental results show that existing streaming omnimodal models still struggle on this benchmark, particularly when handling audio queries compared to text queries. For example, the content accuracy of Qwen3.8-Omni Flash Realtime drops from 64.5% with text queries to 42.8% with audio queries. The authors also fine-tune a Qwen2.5-Omni model using their proposed training dataset and demonstrate improvements in performance.

Critical strengths
  1. The paper constructs a large-scale training dataset and an evaluation benchmark for streaming audio-visual question answering.
  2. As the authors highlight in the related work, the proposed benchmark provides a more fine-grained evaluation by covering a broader range of subtasks compared to existing benchmarks. In addition, the authors aim to distribute query timestamps throughout the video, rather than placing queries primarily at the beginning.
  3. The paper is well-organized and easy to follow.
Critical weaknesses

I think there are several aspects of this paper that could be improved.

  1. Lack of human evaluation for data quality. The authors use AI models and other automated methods for quality control. However, for a benchmark paper, I believe it is still necessary to include human evaluation on at least a subset of the data. For example, human annotators could review the generated questions and answers to verify their correctness. This would help assess whether the automatically generated data and the overall construction pipeline are trustworthy.

  2. Limited distinction from existing benchmarks and lack of new insights. Several existing works, such as OmniInteract (Lu et al., 2026), ROMA (Tian et al., 2026), OmniAssistBench (Sun et al., 2026), and ProAssist (2025), have already explored similar streaming audio-visual interaction settings. I understand that the authors claim to provide a more fine-grained evaluation with a broader range of subtasks compared to existing benchmarks. However, I do not see clear new insights emerging from this new benchmark setting. It would be helpful to demonstrate what additional challenges or limitations of existing models can be revealed by the proposed benchmark but not by previous ones.

  3. Lack of direct comparison with existing training datasets. A more controlled comparison would help demonstrate the value of the proposed training dataset. For example, the authors could fine-tune the same backbone model using NativeQuery-Train and another existing training dataset (e.g., ROMA's training dataset), respectively, and compare their performance under the same evaluation settings. This would help clarify whether the proposed dataset provides additional benefits over existing training datasets.

Overall, I think the paper would benefit from more experimental comparisons and deeper analysis to demonstrate the value of the newly constructed benchmark. Simply introducing a larger benchmark with more subtasks is not sufficient without showing what new insights or capabilities it enables.

My main concerns can be summarized from two perspectives:

  1. Benchmark quality and reliability: Is the benchmark trustworthy? How reliable are the automatically generated questions, answers, and annotations?

  2. Benchmark value and necessity: What additional value does this benchmark provide compared to existing works? Does it reveal new insights into the limitations of current streaming models, or enable new capabilities through its proposed training dataset?

Questions for authors
  1. Benchmark Quality: Could the authors clarify whether they can provide human evaluation results? In particular:

    • (a) Have human annotators independently reviewed a subset of the benchmark to verify the quality of the generated annotations?
    • (b) Could the authors conduct a human evaluation on a subset of the QA tasks and report the results, particularly regarding answer correctness, evidence grounding, and response timing?
  2. Novelty and Insights: Beyond the broader task coverage and more distributed query timestamps, what new insights does NativeQuery provide compared to existing benchmarks? Are there any specific model limitations or failure cases that NativeQuery reveals but existing benchmarks cannot adequately capture?

  3. Training Data Effectiveness: Could the authors provide a more controlled comparison using the same backbone model fine-tuned on different training datasets under comparable training settings? This would help clarify whether the performance improvements come from the proposed dataset itself or simply from additional fine-tuning.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 59qq (Part 1 / 3)

We sincerely thank the Reviewer 59qq for the thorough and constructive review. We appreciate the reviewer’s recognition that NativeQuery goes beyond existing benchmarks with a more fine-grained evaluation, covering a broader range of subtasks and distributing queries throughout the video. We are also grateful that the reviewer acknowledges its accompanying training dataset for streaming audio-visual question answering. We respond to the questions raised by the reviewer point by point below.

W1.1 & Q1. Independent Human Review of Benchmark Annotations.

A1. Thank you for raising this concern. We would like to clarify that human validation is part of our NativeQuery-Bench construction pipeline. It takes place after MLLM-based verification. Ten English-proficient annotators participate in this process. Each sample is independently reviewed by two annotators. A third annotator resolves any disagreements.

The annotators review the audio-visual streams to verify answer correctness, evidence grounding, and query/response timing. They check whether the answers are supported by the source evidence. They also check whether the annotated timestamps are consistent with evidence availability and task requirements. Problematic annotations are corrected or discarded. We have clarified these details in Appendix B of the revised paper.

W1.2 & Q1. Human Evaluation on Answer Correctness, Evidence Grounding, and Response Timing.

A2. We thank the reviewer for this suggestion. We report the verification results across all five tasks. Of the 5,000 initial candidates, 3,449 pass MLLM-based verification. After human validation, 2,735 samples are retained. This shows that human review provides an additional quality check beyond automated verification. We have included the detailed verification statistics in Appendix B of the revised paper.

StageGrdReConAmbPerExeDynManSitIntTotal
Before MLLM verification1,0001,0001,0001,0001,0005,000
After MLLM verification7217887036895483,449
After human validation6007306006002052,735

We further assess the reliability of our automatic content evaluator, GPT-5.6-sol. We sample 300 timely model responses, with 60 from each task. Two annotators independently judge each response as correct or incorrect. A third annotator resolves any disagreements. The evaluator scores responses from 0 to 5. Scores of 3 or higher are considered correct. We compare its judgments with the final human labels. The evaluator achieves an agreement rate of 89.3% and a Cohen's κ of 0.75, with substantial agreement across all five tasks. These results show that the evaluator's judgments are largely consistent with human judgments.

TaskNAgreement (%) ↑Cohen's κ ↑
GrdRe6091.70.81
ConAmb6090.00.80
PerExe6091.70.73
DynMan6086.70.73
SitInt6086.70.70
Overall30089.30.75

W2.1 & Q2. Distinction from Existing Benchmarks.

A3. We thank the reviewer for this comment. NativeQuery differs from existing benchmarks not simply in task coverage, but in the interaction challenges it evaluates.

(1) Compared with OmniInteract, NativeQuery places greater emphasis on cross-modal reasoning and dynamic user intent. It verifies that both audio and visual evidence are necessary to answer each cross-modal question. It also includes 320 forbidden response events to evaluate whether models can suppress responses to outdated requests after users change their intent.

(2) Unlike ROMA, which uses synthesized query voices that differ from the original video speakers, NativeQuery uses voice matching with the original speakers. This reduces obvious changes in voice that models could otherwise use to detect queries.

(3) Compared with OmniAssistBench, which uses segmented videos and offline inference, NativeQuery preserves continuous audio-visual streams and evaluates models through native online inference.

(4) Unlike ProAssist, which provides user goals and follow-up requests as text, NativeQuery embeds spoken requests directly into the audio stream. Models must recognize these requests while processing the ongoing audio-visual input.

AuthorsOpenReview ↗

Response to Reviewer 59qq (Part 2 / 3)

W2.2 & Q2. New Insights and Model Limitations or Failure Cases.

A4. NativeQuery reveals three important limitations that are not captured by existing evaluation settings.

First, explicit-query performance can hide failures in spoken-query understanding. We compare identical questions in text and audio forms. Existing models show substantial performance drops when queries are embedded in continuous audio-visual streams. NativeStream significantly reduces this gap. This demonstrates that understanding spoken requests within a stream remains a distinct challenge.

ModelText C (%) ↑Audio C (%) ↑
Qwen3.8-Omni Flash R64.542.8
NativeStream-7B66.863.2

Second, strong grounded reasoning does not guarantee reliable handling of ambiguity and changing user intent. Model performance varies substantially across these tasks. MiniCPM-o outperforms Qwen3.5-Omni on GrdRe but performs worse on DynMan. NativeStream achieves much larger improvements on ConAmb and DynMan than on GrdRe. These results show that grounded QA performance alone cannot reflect a model's ability to manage evolving user requests.

ModelGrdRe J (%) ↑ConAmb J (%) ↑DynMan J (%) ↑
Qwen3.5-Omni Plus R22.012.511.3
MiniCPM-o-4.5-9B44.519.78.1
NativeStream-7B45.236.725.3

Third, correct response timing does not guarantee reliable interaction. ROMA-7B achieves slightly higher timing accuracy than MiniCPM-o, but its timing precision and content accuracy are much lower. This shows that models may respond at the expected times but still produce incorrect or unnecessary responses.

ModelT Acc. (%) ↑T Prec. (%) ↑C (%) ↑J (%) ↑JF1 (%) ↑
ROMA-7B49.911.19.95.22.0
MiniCPM-o-4.5-9B47.225.454.826.718.4
NativeStream-7B57.830.163.236.224.7

W3.1 & Q3. Controlled Comparison with Existing Training Datasets.

A5. We thank the reviewer for this suggestion. We would like to clarify that ROMA's publicly released training data are incomplete. The public release contains only the proactive subset, while the Reactive QA subset is not available. Fine-tuning on the released data would therefore not reproduce ROMA's full training setup.

Training SubsetOriginal ROMAPublic Release
Reactive QA540,538Not released
Event-Driven Alert27,20727,207
Real-Time Narration108,986108,986
Total676,731136,193

Nevertheless, ROMA-7B and NativeStream-7B share the same Qwen2.5-Omni-7B backbone. As reported in Table 1 of our paper, we evaluate both models on NativeQuery-Bench using the same Streaming Mode setting. NativeStream-7B substantially outperforms ROMA-7B across all four metrics. This comparison demonstrates the stronger performance of our trained model on in-stream spoken-query interaction.

ModelT ↑C ↑J ↑JF1 ↑
ROMA-7B49.99.95.22.0
NativeStream-7B57.863.236.224.7

To separate the contribution of NativeQuery-Train from the effect of additional fine-tuning, we also train a text-query control. It uses the same backbone, NativeQuery-Train samples, and training settings as NativeStream-7B. The only difference is that spoken queries are replaced with text, so both models receive the same amount of fine-tuning on the same data. We evaluate both models in Streaming Mode with text and spoken queries.

Training QueryTest QueryT ↑C ↑J ↑JF1 ↑
TextText63.769.644.029.8
TextSpoken34.241.614.69.2
SpokenText59.166.839.226.3
Spoken (NativeStream-7B)Spoken57.863.236.224.7

The control performs well on text queries (44.0% J), but reaches only 14.6% J on spoken queries, compared with 36.2% for NativeStream-7B. This shows that the improvement comes from the in-stream spoken-query supervision in NativeQuery-Train, not from additional fine-tuning alone.

AuthorsOpenReview ↗

Response to Reviewer 59qq (Part 3 / 3)

W3.2 & Q3. Training Data Effectiveness vs. Additional Fine-Tuning.

A6. We further compare NativeStream-7B with the original Qwen2.5-Omni-7B to assess the effectiveness of NativeQuery-Train. Since the base model cannot autonomously decide when to respond, we evaluate both models in Oracle Mode. Both receive the same audio-visual inputs and follow the same evaluation protocol.

Fine-tuning on NativeQuery-Train substantially improves the backbone's ability to understand spoken requests embedded in continuous audio-visual streams. The fine-tuned model also exceeds the base model's text-query accuracy. Together with the ROMA comparison, these results support the effectiveness of NativeQuery-Train.

ModelQuery TypeC (%) ↑
Qwen2.5-Omni-7BText62.0
Qwen2.5-Omni-7BAudio36.7
NativeStream-7BAudio63.9
Reviewer_ZLsqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on streaming QA and introduces NativeQuery-Train, NativeQuery-Bench, and fine-tuned two NativeStream models with the training set. The benchmark covers grounded reasoning, ambiguity and conflict handling, persistent execution, dynamic task management, and situated interaction. The benchmark results show that existing models often separate correct content from correct timing, while the trained NativeStream model improves their joint accuracy and transfers to OmniInteract.

Critical strengths
  1. Placing spoken requests inside the video stream is meaningful, as most bechmarks would only provide a separate text query at a known time. The task setup in this paper better captures the design of a streaming system where it needs to recognize, revise, or resume user requests.

  2. The paper provides both training data and a benchmark, then shows improvements in joint timing-and-content accuracy after training with the dataset. The improvements have also shown to transfer to OmniInteract. This makes the paper's contribution useful for both evaluation and model development.

Critical weaknesses
  1. The spoken queries are synthesized and mixed into existing videos. Synthetic voices can be regular-sounding and inputted at timings that the models could expect. These together might make query detection easier. Real human recordings often have more inturruptions, stutter, and connecting sounds. So the synthetized audio might not capture spontaneous speech in real conversations. This is critical because realism is the main motivation of the paper.

  2. The timing metric uses a fixed three-second tolerance, and its content is judged by an LLM. I wonder how robust this evaluation is? Say if it's judged by human, different people might chose different timing as well but would likely come to a concensus. Without stronger analysis between human concensus and the judge's agreement, it is hard to know how stable the reported joint gains are.

Questions for authors
  1. How much performance changes on human-recorded queries, unseen speakers/TTS systems, or more realistic overlapping speech? A small controlled result would give me a better understanding on how to interpret benchmark.

  2. Can you report whether the model ranking is stable under tighter timing windows and also the judge model's agreement with human judges?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZLsq (Part 1 / 2)

We sincerely thank the Reviewer ZLsq for the valuable feedback and the positive assessment. We appreciate the reviewer’s recognition that placing spoken requests inside the video stream is meaningful and that NativeQuery benefits both evaluation and model development. We respond to the questions raised by the reviewer point by point below.

W1.1 & Q1. Human-recorded queries and unseen speakers/TTS systems.

A1. As described in Sec. 3.2, NativeQuery uses source-matched TTS, task-dependent query timestamps, and loudness matching to reduce obvious acoustic differences and predictable query positions.

We further evaluate NativeStream-7B on 100 samples from NativeQuery-Bench, with 20 samples from each of the five tasks. We replace the synthesized speech with human recordings of the same questions, while keeping the source videos, query content, and evaluation protocol unchanged. Human-recorded speech improves timing accuracy from 58.2% to 63.4% and joint accuracy from 36.5% to 40.0%, while content accuracy remains nearly unchanged. Notably, the human recordings have more distinctive speaker characteristics than our source-matched TTS queries, making them easier to distinguish from the original video speech. These results suggest that our synthesized queries do not artificially simplify query detection through obvious speaker differences.

Query SpeechT ↑C ↑J ↑JF1 ↑
Synthesized Speech (Ours)58.262.836.524.9
Human-recorded Speech63.463.140.027.5
Δ+5.2+0.3+3.5+2.6

To further test unseen speakers and TTS systems, we evaluate on OmniInteract, whose 1Q1A subset contains human-recorded queries and whose 1QnA subset uses a different TTS system. As reported in Table 2 of the main paper, NativeStream-7B achieves the best results on both subsets, as shown below. These results demonstrate that NativeStream generalizes to unseen speakers and synthesized queries.

Model1Q1A Global ↑1QnA ↑
AURA0.4670.052
MiniCPM-o-4.5-9B0.4560.015
NativeStream-3B0.4500.047
NativeStream-7B0.5270.059

W1.2 & Q1. More Realistic Overlapping Speech.

A2. We further evaluate NativeStream-7B under overlapping speech using the same 100 samples as in A1. We keep the source videos and queries unchanged, but insert the queries where the original videos contain speech. As shown below, both speech types perform substantially worse than in A1. Joint accuracy drops from 36.5% to 10.3% for synthesized speech and from 40.0% to 10.4% for human-recorded speech. The similar results suggest that overlapping speech is a major challenge for NativeStream, regardless of whether the queries are synthesized or human-recorded.

Query SpeechT ↑C ↑J ↑JF1 ↑
Synthesized Speech27.437.610.36.8
Human-recorded Speech28.736.210.46.7

W2.1 & Q2. Model Ranking under Tighter Timing Windows.

A3. We thank the reviewer for this suggestion. To assess the robustness of our evaluation, we evaluate all models under five timing tolerances of 1, 2, 3, 5, and 10 seconds. We report overall joint accuracy (J) and joint F1 (JF1). NativeStream-7B achieves the highest J and JF1 under the 2-, 3-, 5-, and 10-second tolerances, while NativeStream-3B leads under the stricter 1-second tolerance. NativeStream-3B has lower inference latency (RTF: 0.337 vs. 0.369; TTFT: 287 vs. 320 ms), which helps it meet stricter timing requirements. In contrast, NativeStream-7B shows larger gains as the tolerance increases from 1 to 3 seconds. This suggests that more of its correct responses fall outside the 1-second window. Although the ranking changes under the strictest tolerance, NativeStream achieves the best performance across all five timing windows. These results show that our main conclusions do not depend on the specific 3-second tolerance. We have added this analysis to Appendix E of the revised paper. Each cell reports J/JF1 (%).

Model1s2s3s5s10s
Qwen3.0-Omni Flash R6.8/2.215.7/5.220.5/6.822.1/7.422.7/7.6
Qwen3.5-Omni Plus R6.0/2.216.4/5.924.6/8.928.8/10.430.0/10.9
Qwen3.8-Omni Flash R4.2/1.512.6/4.521.3/7.627.4/9.828.5/10.3
Gemini 2.5 Flash Live3.5/1.69.8/4.315.1/6.718.3/8.118.8/8.5
Gemini 3.8 Live4.8/2.211.1/5.315.7/7.518.1/8.718.9/9.1
ROMA-7B1.7/0.63.8/1.45.2/2.06.0/2.36.2/2.4
MiniCPM-o-4.5-9B8.9/6.119.7/13.526.7/18.430.0/20.730.8/21.3
NativeStream-3B9.7/6.722.6/15.730.6/21.333.9/23.634.6/24.2
NativeStream-7B8.2/5.623.8/16.236.2/24.741.6/28.542.5/29.1
AuthorsOpenReview ↗

Response to Reviewer ZLsq (Part 2 / 2)

W2.2. Human Consensus on Response Timing.

A4. We thank the reviewer for raising this concern. We would like to clarify that human validation is part of our NativeQuery-Bench construction pipeline and is conducted after MLLM-based verification. We conduct human validation of NativeQuery-Bench with 10 English-proficient annotators. Each sample is independently reviewed by two annotators, with disagreements resolved by a third. They examine the audio-visual streams to verify whether the query and response timestamps are consistent with the available evidence and task requirements. Incorrect timestamps are corrected, and samples that cannot be corrected are discarded. This process ensures that the final timestamps undergo human verification. The validation results are shown below. We have added these details to Appendix B of the revised paper.

StageGrdReConAmbPerExeDynManSitIntTotal
Before MLLM verification1,0001,0001,0001,0001,0005,000
After MLLM verification7217887036895483,449
After human validation6007306006002052,735

W2.3 & Q2. LLM Judge's Agreement with Human Judges.

A5. We thank the reviewer for this suggestion. To assess the reliability of our automatic evaluator, GPT-5.6-sol, we sample 300 timely model responses, with 60 from each task. Two annotators independently judge each response as correct or incorrect. A third annotator resolves any disagreements. The annotators use the same query, reference answer, and model response provided to the evaluator. The evaluator scores each response from 0 to 5. A score of 3 or higher is considered correct. We compare its judgments with the final human labels using agreement rate and Cohen's κ. The evaluator achieves an agreement rate of 89.3% and a Cohen's κ of 0.75, with substantial agreement across all five tasks. These results show that the evaluator's judgments are largely consistent with human judgments.

TaskNAgreement (%) ↑Cohen's κ ↑
GrdRe6091.70.81
ConAmb6090.00.80
PerExe6091.70.73
DynMan6086.70.73
SitInt6086.70.70
Overall30089.30.75

Thank you for providing the details on the judge construction and more evaluation details. The supporting evidences helped clarify my doubts. It's quite clear that the proposed dataset, both for the benchmark and for training, have merits that could be beneficial to the community. With that said, I can see why reviewer 9UnP has concerns on novelty, especially comapred to OmniInteract, and that it would be more impactful to share insights that have not been readily available in other papers. To improve the paper's presentation, maybe consider making the ability to model "evolving unser intent" a more central contribution of the paper. I recommend revising the paper, but want to acknowledge that the benchmark is well constucted and shows promising training results.

AuthorsOpenReview ↗

Follow-up Response to Reviewer ZLsq

We thank the reviewer for the positive feedback and for acknowledging that our responses have addressed the concerns. We appreciate the constructive suggestion and will refine the presentation of our contributions in the revised paper, with greater emphasis on evaluating how models handle evolving user intent.

Reviewer_9UnPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces NativeQuery, consisting of NativeQuery-Train, a large-scale training corpus, and NativeQuery-Bench, an evaluation benchmark for spoken-query interaction in continuous audio-visual streams. NativeQuery-Train contains 200K user-query turns, while NativeQueryBench contains 3,419 evaluation user-query turns across five task families and fifteen subtasks, covering grounded multimodal reasoning, conflict and ambiguity handling, persistent task execution, dynamic task management, and situated interaction reasoning. NativeQuery integrates queries into the original streams while preserving their native speakers and surrounding audio-visual context. The authors further introduce NativeStream, a streaming omnimodal model trained on NativeQueryTrain for such in-stream interaction. Experiments on NativeQuery-Bench reveal a substantial gap between explicit text queries and in-stream spoken queries, with Qwen 3.8 Omni Flash Realtime dropping from 64.5% to 42.8% in content accuracy. NativeStream substantially mitigates this problem, with NativeStream-7B achieving 36.2% joint accuracy on NativeQuery-Bench and 0.419 All-Global on OmniInteract, outperforming existing streaming omnimodal models.

Critical strengths
  1. A large training with evaluation dataset. NativeQuery provides both 200K training data and 3,419 evaluation data, excelling previous works such as OmniInteract and ROMA.

  2. The four-stage data pipeline extracts and connects temporally localized visual, speech, sound, and text evidence to generate task-specific queries, effectively preserving temporal provenance while distinguishing the authored query time , the evidence-sufficiency time, and the realized query-audio boundaries.

  3. Experiments on self-constructed benchmarks and OmniInteract prove the effectiveness of the proposed data and model.

Critical weaknesses
  1. The authors admit that "OmniInteract is more closely aligned with our setting, preserving continuous audio-visual input and evaluating models through native online inference". I think the difference in setting between OmniInteract and NativeQuery should be discussed more in detail as the difference seems more in scope covering and annotation granularity .

  2. The novelty in model architecture design and inference pipeline seems limited. The paper does not provide inference statistics such as RTF, TTFT, which reflects the practical usage of the model in real-world scenarios.

  3. The training data come from instruction videos, e.g., OmniVideo-100K、COIN、YouCook2. The paper lacks more complicated scenarios such as meeting, live show, etc.

  4. The experiments lack comparison with the base model, i.e., Qwen2.5-Omni. This may be an important ablation study.

Questions for authors

I will raise my score if the rebuttal of the authors addresses my concerns.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 9UnP (Part 1 / 2)

We sincerely thank the Reviewer 9UnP for the valuable comments and suggestions. We appreciate the reviewer’s recognition that NativeQuery contributes large-scale data for both training and evaluating streaming omni-modal models. We are also grateful that the reviewer acknowledges our temporally grounded data pipeline and the effectiveness of our data and model. We respond to the questions raised by the reviewer point by point below.

W1. Differences between OmniInteract and NativeQuery.

A1. Thank you for raising this question. OmniInteract and NativeQuery both evaluate continuous audio-visual interaction through native online inference. Our contribution is not a new inference protocol. Instead, NativeQuery focuses on two requirements for valid responses: tracking the user's current intent and grounding responses in available multimodal evidence.

1. Evolving User Intent. OmniInteract evaluates interruption and resumption. In these interactions, the original request remains active and must be resumed after an interruption. NativeQuery additionally evaluates Request Revision (ReqRev), where a later request replaces the original one. The model must suppress responses to the outdated request, even when its original trigger occurs later. NativeQuery-Bench contains 320 such superseded response events. It also evaluates conflict and ambiguity handling, where models must correct false premises or ask clarifying questions rather than provide unsupported answers. These tasks go beyond interruption handling. They test whether models can update their active tasks and respond according to the user's latest intent.

2. Evidence-Grounded Responses. OmniInteract annotates response triggers, valid response windows, target answers, and the earliest valid response time. NativeQuery further develops an evidence-driven construction pipeline. It extracts and links specific visual, speech, sound, and text observations, together with their relations and verified availability times, to each response event. NativeQuery-Bench contains 4,331 response annotations linked to 9,221 evidence items. This provides more explicit evidence-level supervision. It allows us to evaluate not only when a response becomes valid, but also which multimodal evidence supports it and when the complete evidence chain becomes available.

These distinctions matter because strong grounded reasoning does not necessarily imply effective interaction management. To illustrate this, we reproduce the relevant results from Table 1 of the paper below. NativeStream-7B improves conflict and ambiguity handling and dynamic task management while maintaining comparable performance on grounded reasoning.

ModelGrdRe J (%) ↑ConAmb J (%) ↑DynMan J (%) ↑
Qwen3.5-Omni Plus R22.012.511.3
MiniCPM-o-4.5-9B44.519.78.1
NativeStream-7B45.236.725.3

Unlike OmniInteract, which serves as an evaluation benchmark, NativeQuery also provides NativeQuery-Train with approximately 200K spoken-query turns. This enables training and evaluation under a unified task design.

W2. Model novelty and inference efficiency.

A2. We thank the reviewer for this comment. We would like to clarify that our primary contribution is not a novel model architecture or inference pipeline. Instead, we demonstrate that in-stream spoken-query supervision from NativeQuery-Train helps existing omnimodal models generate more accurate and timely responses in continuous audio-visual streams.

We additionally measure RTF and TTFT to evaluate inference efficiency. All experiments are conducted on a single NVIDIA A800-SXM4-80GB GPU with batch size 1. RTF is defined as the model computation time divided by the video duration. TTFT measures the time from when the input at response onset is fully received to the generation of the first text token. Compared with MiniCPM-o-4.5-9B, both NativeStream models achieve higher J scores, lower RTFs, and lower TTFTs. Their RTF values are both below 1, indicating that they can process video faster than real time under this setting.

ModelJ (%) ↑RTF ↓TTFT (ms) ↓
MiniCPM-o-4.5-9B26.70.522345
NativeStream-3B30.60.337287
NativeStream-7B36.20.369320
AuthorsOpenReview ↗

Response to Reviewer 9UnP (Part 2 / 2)

W3. Coverage of non-instructional scenarios.

A3. We thank the reviewer for this comment. While COIN and YouCook2 are instructional datasets, OmniVideo-100K is an open-domain collection of English YouTube videos. It covers news, sports, interviews, talk shows, press conferences, hearings, and entertainment. Among the 424 videos with available YouTube category metadata, only 0.9% are labeled Howto & Style (YouTube's category for instructional and tutorial videos).

We use scene annotations to count query turns from videos containing meeting rooms, hearing rooms, press briefings, and show venues. The results show that both NativeQuery-Train and NativeQuery-Bench cover these non-instructional scenarios, including the meeting and show settings.

SettingTrain turnsBench turns
Meeting and hearing rooms7,623162
Press conferences and briefings7,152212
Stages and show venues9,204288

W4. The experiments lack comparison with the base model, i.e., Qwen2.5-Omni. This may be an important ablation study.

A4. We thank the reviewer for this suggestion. Since Qwen2.5-Omni cannot decide when to respond on its own, we compare it with NativeStream-7B in Oracle Mode. In this mode, inference is triggered at annotated response times rather than by the model. Therefore, only content accuracy (C) can be evaluated.

The results for Qwen2.5-Omni-7B with text and audio queries are reported in Table 3 of the original paper. We additionally evaluate NativeStream-7B with audio queries under the same setting. NativeStream-7B receives the same audio-visual input as Qwen2.5-Omni-7B, with spoken queries embedded in the audio. Qwen2.5-Omni-7B performs substantially worse with audio queries than with text queries. After fine-tuning on NativeQuery-Train, content accuracy with audio queries increases from 36.7% to 63.9%. These results demonstrate that NativeQuery-Train substantially improves the model's ability to understand spoken queries embedded in continuous audio-visual streams.

ModelQuery TypeC (%) ↑
Qwen2.5-Omni-7BText62.0
Qwen2.5-Omni-7BAudio36.7
NativeStream-7BAudio63.9

Thanks for the additional experiments. The latency results and oracle comparison with the base model address those concerns, but my novelty concern remains. Request revision is highlighted as a key distinction, yet the reported DynMan scores combine it with interruption/resumption, without separately showing how well models suppress responses to outdated requests. Likewise, the text-versus-spoken control shows that spoken-query training helps, but does not isolate the benefits of the proposed task design or evidence-driven construction. I acknowledge the dataset’s usefulness, but remain unconvinced that the contribution beyond broader coverage and scale is sufficiently demonstrated. I will keep my score of 2.

Reviewer_gwSFOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies streaming omni-modal interaction where the user's spoken query is part of the continuous audio-visual stream instead of a separate text turn. It contributes three things:

  • Data and Benchmark. NativeQuery-Train has about 160K dialogue records, 224K spoken query turns and 11K source videos. NativeQuery-Bench has 3,419 query turns, 4,331 response annotations and 2,050 videos of 56–180 s. Both cover five task families (15 subtasks): grounded reasoning, conflict and ambiguity, persistent tasks, request revision and interruption, and situated social reasoning.

  • Construction Pipeline. Gemini 3.7 Flash extracts time-stamped multimodal evidence and GPT-5.6-sol proposes and verifies QA items over an evidence graph. Each query is then synthesized with Qwen3-TTS, in a voice cloned from a speaker in the video, and mixed into the original audio track.

  • Model. NativeStream-3B and NativeStream-7B is Qwen2.5-Omni fine-tuned with response-time supervision.

Critical strengths

S1. The problem matters. An always-on streaming assistant has to notice and interpret requests that arrive inside the same audio channel as scene speech and sounds. Most streaming benchmarks deliver the query out-of-band.

S2. There is a training set. The relevent benchmark, OmniInteract, has no training data. The paper also shows some transfer to OmniInteract, mainly on its proactive and nested splits.

S3. The task design goes beyond single-shot QA. It includes one standing request with multi-run responses, request revision with explicit forbidden responses, interruption and resumption, and clarification. Evidence is annotated with timestamps and modality, so the items can be audited. The authors released the annotations and the scorer, which made the checks below possible.

Critical weaknesses

W1. Some annotations do not support the intended task and look like wrong.

  1. In 42/320 revision examples, answers are forbidden before the user changes the request. For example, D_D1_0012_eval forbids an answer at 44.0 s, but the revision starts at 44.42 s.
  2. In 92/280 interruption examples, the first answer is expected before the interruption, so the model need not resume an unfinished task.
  3. Only 133/1512 PerExe answers cite both audio and visual evidence. And how many actually need both is unclear.

And No human audit is reported for NativeQuery-Bench. The authors should report agreement between human judgments and the automatic evaluator, and release the data, code, and model checkpoints.

W2. The metrics need clearer definitions and validation.

  1. Within each task family, C measures content correctness conditional on a timing match, and J = T × C. The C–J gap is therefore partly built into the definitions. T and J do not directly penalize extra unmatched outputs, although forbidden violations affect J. This matters given the API models' 9–13% time precision in Table 5.

  2. The scorer also defaults to allowing up to 10 seconds of lateness for required responses, capped by the next target, whereas the paper describes a three-second tolerance. Is allowing up to 10 seconds to respond too generous for real-time interaction? If the model can wait that long, could it just buffer a clip and answer it in one pass? The paper says three seconds, so which setting was used?

  3. joint F1 should be reported simultaneously in the main result. It not only checks whether the model answers correctly at the right time, but also penalizes redundant answers. This way, it can distinguish between "saying the right thing when it should be said" and "saying it everywhere and getting some right by chance". And the paper does not report <u>how well the judge agrees with human ratings</u>.

W3. The baseline model comparison is missing.

  1. Qwen2.5-Omni, the baseline model for NativeStream, is absent from the main streaming comparisons. The reported OmniInteract increase from 0.368 to 0.419 compares MiniCPM-o with NativeStream. But it is not a before-and-after training comparison.
  2. Video-level separation between training and benchmark data is also unspecified.
  3. Beyond one epoch and base-model defaults, essential details such as learning rate, batch size, and silence- and forbidden-response supervision are missing.

W4. The speech setup needs a control. Both datasets use synthetic queries. OmniInteract shows some transfer, but testing the same NativeQuery questions with human speech would help check whether the results depend on TTS or mixing. The code also gives each task family its own system prompt and I think this could be disclosed and tested against a shared prompt.

BTW, Some claims do not match the tables. NativeStream-7B does not lead J in every family: PerExe is 43.1 vs. 43.2, and SitInt is 30.7 vs. 34.1. Appendix B's claim that it leads every timing metric in every family also needs correction.

Questions for authors
  1. How have the authors checked the quality of the labels and timing? Is a human audit planned, especially for the revision and interruption cases noted above?
  2. Please confirm the timing windows and judge thresholds used in the tables, and better provide the report of joint F1 under 1/3/5/10-second tolerances.
  3. Please report the streaming Qwen2.5-Omni baseline and a text-query training control, and clarify whether training and test videos overlap.
  4. Please define Oracle Mode and explain how text queries were delivered to MiniCPM-o and NativeStream.
7 comments
AuthorsOpenReview ↗

Response to Reviewer gwSF (Part 1 / 4)

We sincerely thank the Reviewer gwSF for the valuable comments and suggestions. We appreciate the reviewer’s recognition that NativeQuery tackles a meaningful problem and that its task design goes beyond single-shot QA, including request revision, interruption and resumption, and clarification. We are also grateful that the reviewer acknowledges our training set, which addresses the scarcity of training data for in-stream spoken queries. We respond to the questions raised by the reviewer point by point below.

W1.1 & W1.2. Clarification of query timestamps in revision and interruption tasks.

A1. We thank the reviewer for carefully checking the annotations. We clarify that question_time records the final query timestamp used for dataset construction and evaluation. The question_start and question_end are outdated keys from earlier annotation stages. The reported 42/320 revision and 92/280 interruption cases arise from using these outdated keys instead of question_time. Therefore, these cases do not indicate errors in the final annotations. We will remove the outdated keys to avoid confusion.

W1.3. Necessity of Audio-Visual Evidence in PerExe.

A2. We appreciate the reviewer's careful examination of the annotations. Among the 1,512 PerExe responses, 133 are cross-modal responses that require both audio and visual evidence. As described in Sec. 3.2, every cross-modal candidate goes through a modality-necessity check: we remove either the audio or the visual evidence and verify that the question can no longer be answered. Cross-modal candidates that can still be answered with only one modality are discarded. As a result, all 133 cases have passed this check and require both modalities, rather than merely citing both. We will clarify this verification process in Appendix B.

W1.4 & Q1. Human Validation of Benchmark Annotations.

A3. We thank the reviewer for this suggestion. We would like to clarify that human validation is part of our NativeQuery-Bench construction pipeline and is conducted after MLLM-based verification. A total of 10 English-proficient annotators take part in this process. Each sample is independently checked by 2 annotators. A third annotator resolves any disagreements. They review the audio-visual streams to check whether the answers are correct, supported by the evidence, and given at the right time. Samples with errors are first corrected where possible. Those that cannot be corrected are discarded. The validation results are shown below. We have added these details to Appendix B of the revised paper.

StageGrdReConAmbPerExeDynManSitIntTotal
Before MLLM verification1,0001,0001,0001,0001,0005,000
After MLLM verification7217887036895483,449
After human validation6007306006002052,735

W1.5. Human–LLM Judge Agreement.

A4. We thank the reviewer for this suggestion. To assess the reliability of our automatic evaluator, we sample 300 timely model responses, with 60 from each task. Each response is independently checked by 2 annotators. A third annotator resolves any disagreements. The annotators judge whether each response is correct based on the query and reference answer. The automatic evaluator scores each response from 0 to 5. A score of 3 or higher is considered correct. We compare the evaluator's judgments with the final human labels using agreement rate and Cohen's κ.

The evaluator achieves 89.3% agreement with human judgments and a Cohen's κ of 0.75. The results show substantial agreement across all five tasks.

TaskNAgreement (%) ↑Cohen's κ ↑
GrdRe6091.70.81
ConAmb6090.00.80
PerExe6091.70.73
DynMan6086.70.73
SitInt6086.70.70
Overall30089.30.75
AuthorsOpenReview ↗

Response to Reviewer gwSF (Part 2 / 4)

W2.1 & Q2. Metric Definitions and Joint F1.

A5. Thank you for the valuable suggestion. T, C, and J provide complementary views of model performance. T measures whether the model responds at the correct time, C evaluates the correctness of timely responses, and J measures how often both requirements are met. Together, they help distinguish timing failures from content errors. To further account for redundant responses, we additionally report joint precision (J_P) and joint F1 (JF1).

As shown below, NativeStream-7B achieves the best performance across all four metrics, including joint precision and joint F1. This confirms that its advantage is not driven by excessive responses. We have clarified the metric definitions in Sec. 5.1 and Appendix C, added JF1 to the main results in Table 1, and reported additional precision metrics in Table 6 of Appendix D.

ModelT_P↑J ↑J_P ↑JF1 ↑
Qwen 3.0-Omni Flash R9.320.54.26.8
Qwen 3.5-Omni Plus R10.024.65.58.9
Qwen 3.8-Omni Flash R10.421.34.77.6
Gemini 2.5 Flash Live12.415.14.56.7
Gemini 3.8 Live13.215.75.27.5
ROMA-7B11.15.21.22.0
MiniCPM-o-4.5-9B25.426.714.118.4
NativeStream-3B28.830.616.421.3
NativeStream-7B30.136.218.924.7

W2.2. Timing Window Configuration.

A6. We would like to clarify that all main results use --tolerance 3 --tolerance_late_max 3. The 10 s value is an unused default in the evaluation script. We agree that a 10 s delay is too generous for real-time interaction, as it could allow models to buffer video clips before responding. However, this setting is not used in our main experiments. We have clarified the temporal matching protocol in Appendix C.

W2.3 & Q2. Real-Time Validity and Timing Tolerance Robustness.

A7. Thank you for the valuable suggestion. To assess the robustness of our evaluation, we evaluate all models under 5 tolerance windows (δ = 1, 2, 3, 5, 10 s). We report both joint accuracy (J) and joint F1 (JF1).

As shown below, NativeStream-7B achieves the highest J and JF1 under the 2, 3, 5, and 10s tolerances, while NativeStream-3B slightly leads under the stricter 1s tolerance. NativeStream-3B has lower inference latency (RTF: 0.337 vs. 0.369; TTFT: 287 vs. 320 ms), which helps it meet stricter timing requirements. In contrast, NativeStream-7B shows larger gains as the tolerance increases from 1 to 3 s. This indicates that more of its correct responses fall outside the 1 s window. Nevertheless, NativeStream-7B consistently outperforms all other models under the 2, 3, 5, and 10 s tolerances. These results show that our main conclusions do not depend on the specific 3s tolerance. We have added this sensitivity analysis to Appendix E. Each entry reports J / JF1 (%).

Model1 s2 s3 s5 s10 s
Qwen 3.0-Omni Flash R6.8/2.215.7/5.220.5/6.822.1/7.422.7/7.6
Qwen 3.5-Omni Plus R6.0/2.216.4/5.924.6/8.928.8/10.430.0/10.9
Qwen 3.8-Omni Flash R4.2/1.512.6/4.521.3/7.627.4/9.828.5/10.3
Gemini 2.5 Flash Live3.5/1.69.8/4.315.1/6.718.3/8.118.8/8.5
Gemini 3.8 Live4.8/2.211.1/5.315.7/7.518.1/8.718.9/9.1
ROMA-7B1.7/0.63.8/1.45.2/2.06.0/2.36.2/2.4
MiniCPM-o-4.5-9B8.9/6.119.7/13.526.7/18.430.0/20.730.8/21.3
NativeStream-3B9.7/6.722.6/15.730.6/21.333.9/23.634.6/24.2
NativeStream-7B8.2/5.623.8/16.236.2/24.741.6/28.542.5/29.1

W2.4. Judge Thresholds and Human–LLM Agreement.

A8. Thank you for the valuable suggestion. Our automatic evaluator, GPT-5.6-sol, scores each timely response from 0 to 5. A score of 3 or higher is considered correct. We use this threshold consistently across all reported results.

To validate the evaluator, we compare its judgments with human annotations on 300 timely model responses. The evaluator achieves 89.3% agreement and a Cohen's κ of 0.75, showing substantial agreement with human judgments. The detailed experimental results and analysis are presented in our response to W1.5 (A4).

AuthorsOpenReview ↗

Response to Reviewer gwSF (Part 3 / 4)

W3.1 & Q3. Qwen2.5-Omni Baseline Comparison.

A9. We thank the reviewer for raising this concern. We would like to clarify that the reported OmniInteract results compare NativeStream-7B (0.419) with the previous SOTA model, MiniCPM-o-4.5-9B (0.368), rather than its base model, Qwen2.5-Omni-7B.

Since Qwen2.5-Omni cannot autonomously decide when to respond, it cannot be directly evaluated in Streaming Mode. We therefore compare Qwen2.5-Omni-7B and NativeStream-7B in Oracle Mode. In this setting, inference is triggered at the annotated response time. Both models receive the same audio-visual input up to each annotated response time, with spoken queries mixed into the audio.

These results demonstrate that fine-tuning on NativeQuery-Train substantially improves the base model's ability to understand spoken queries embedded in continuous audio-visual streams. NativeStream can also proactively respond to spoken queries in streaming mode.

ModelEvaluation ModeT ↑C ↑J ↑JF1 ↑
Qwen2.5-Omni-7BOracle—36.7——
NativeStream-7BOracle—63.9——
NativeStream-7BStreaming57.863.236.224.7

W3.2 & Q3. Text-Query Training Control.

A10. We thank the reviewer for this suggestion. We additionally train a text-query control using the same Qwen2.5-Omni-7B backbone and NativeQuery-Train samples as NativeStream-7B. We replace spoken queries with text while keeping the target answers, response-time supervision, and training settings unchanged. We evaluate both models with text and spoken queries in streaming mode.

The text-query control performs well on text queries, confirming that it learns the same tasks. However, its performance drops substantially when evaluated with spoken queries. In contrast, NativeStream-7B maintains strong performance under both conditions. These results show that text-query training alone is insufficient for reliably understanding spoken queries embedded in continuous audio-visual streams.

Training QueryTest QueryT ↑C ↑J ↑JF1 ↑
TextText63.769.644.029.8
TextSpoken34.241.614.69.2
SpokenText59.166.839.226.3
SpokenSpoken57.863.236.224.7

W3.3 & Q3. Training–Test Video Separation.

A11. There is no video-level overlap between NativeQuery-Train and NativeQuery-Bench. We exclude all the training samples from videos used in the benchmark to prevent data leakage.

W3.4. Training Hyperparameters.

A12. We train for one epoch with a learning rate of 1e-5, a per-GPU batch size of 1, and 16 gradient accumulation steps across 8 GPUs (effective batch size 128).

W3.5. Silence and Forbidden-Response Supervision.

A13. As described in Lines 398–400, we follow the autoregressive response-time supervision [1]. We insert </Response> followed by the target answer at each annotated response time. No response targets are provided during silence or forbidden-response intervals.

[1] Xia, Jiaer, et al. "Streaming Video Instruction Tuning." CVPR, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Program Chairs, Senior Area Chairs, Area Chair, and all reviewers for their time and valuable feedback. As the discussion period concludes, we would like to summarize our key contributions, experimental findings, and discussion outcomes below.

1. Key Contributions and Insights

NativeQuery introduces a large-scale training dataset, benchmark, and NativeStream for in-stream spoken-query interaction. Its evidence-driven construction enables the evaluation of response validity, timing, and evolving user intent, revealing important limitations of existing models:

  • In-stream spoken queries: Models struggle with spoken queries despite performing well with text queries (Qwen3.8-Omni Flash Realtime: 64.5% vs. 42.8%).

  • Evolving user intent: Existing streaming benchmarks can overlook failures to handle revised requests. MiniCPM-o achieves 0.599 on OmniInteract Nested (best: 0.610), yet still responds to outdated requests in 28.8% of the 320 superseded cases, compared with 10.6% for NativeStream-7B. On Request Revision, the evaluated baselines achieve at most 5% joint accuracy, compared with 21.9% for NativeStream-7B.

  • Conflict and ambiguity: Accurate grounded reasoning does not guarantee effective conflict resolution or ambiguity clarification. Even with text queries and oracle timing, Gemini 3.1 Pro achieves 89.2% on grounded reasoning but only 72.4% on conflict and ambiguity handling.

2. Experimental Evidence and Clarifications

We addressed the main concerns through the following experiments and clarifications:

  • Benchmark quality (gwSF, 59qq, ZLsq): The 42/320 and 92/280 cases flagged by gwSF arise from outdated keys rather than the finalized question_time used for evaluation. All 133 cross-modal PerExe responses pass a modality-necessity check. We clarified the human validation process involving 10 annotators, with two independent annotators and a third resolving disagreement (Appendix B). The automatic judge achieves 89.3% agreement with human judgments (Cohen's κ = 0.75).

  • Training effectiveness (gwSF, 9UnP, 59qq): NativeStream-7B improves spoken-query accuracy from 36.7% to 63.9% over the base model in Oracle Mode. Under matched training settings, text-query training achieves only 14.6% joint accuracy on spoken queries, compared with 36.2% for NativeStream-7B.

  • Evaluation reliability (gwSF, ZLsq): All main results use a 3 s tolerance. The 10 s script default was not used. Joint precision and F1 confirm that NativeStream-7B's improvements are not driven by redundant responses. Across five timing tolerances (1–10 s), a NativeStream model consistently achieves the best performance.

  • Robustness and efficiency (gwSF, ZLsq, 9UnP): On the human-recorded speech evaluation, joint accuracy improves from 36.5% to 40.0%. A unified system prompt yields comparable performance (36.2% vs. 34.3% J). NativeStream-7B also achieves faster real-time processing than MiniCPM-o (RTF: 0.369 vs. 0.522). Overlapping speech remains challenging for both synthesized and human queries (J ≈ 10%).

3. Discussion Outcomes

  • 9UnP (2 → 3): Confirmed that our latency and base-model comparisons addressed key concerns and raised the rating from 2 to 3. While reservations about novelty remained, the reviewer acknowledged NativeQuery as a useful contribution following our additional analyses on evolving user intent.

  • ZLsq (3): Acknowledged that our responses clarified their concerns, recognized the value of our benchmark and training data, and suggested emphasizing evolving user intent.

  • 59qq (2): Acknowledged that our rebuttal clarified several concerns and moved the assessment closer to borderline. In our follow-up, we provided further empirical evidence of model failures overlooked by existing benchmarks, particularly in Request Revision and conflict and ambiguity handling.

  • gwSF (2): We provided point-by-point responses to all concerns, including annotation quality, evaluation metrics, controlled comparisons, and speech robustness, but have not yet received follow-up feedback.

We have clarified the evaluation protocol, metrics, and validation procedures in the revised manuscript. We will further emphasize the evaluation of evolving user intent and the model limitations revealed by NativeQuery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.