acceptodds
Under review as a conference paper at ICLR 2027

ProactiveAudioBench: Evaluating Condition-Driven Responses in Audio Streams

Abstract

Continuous audio assistance requires models to execute a listening instruction as audio arrives, deciding when a target occurrence warrants a notification and when to remain silent. We introduce **ProactiveAudioBench**, covering sound events, speech, and music through Detection, Tracking, Counting, and Matching. These tasks organize first, repeated, and Nth-occurrence notification rules alongside targets specified by acoustic references. The benchmark contains 1,999 listening requests and 1,851 reference response opportunities, with negative requests requiring silence. Our evaluation framework supports simulated and native streaming, representing each request as a sequence of expected and emitted responses. Joint recall, joint precision, and negative-request false alarms assess the timing and content of notifications together with omissions and unnecessary replies. Across 23 system configurations and four tasks, we find protocol-dependent differences in error timing: simulated-streaming errors cluster before or near reference checkpoints, whereas native-streaming errors show a longer tail of delayed responses.

Primary area:
datasets and benchmarks
Keywords:
Proactive Audio Interaction, Streaming Audio Understanding, Audio-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_y7drOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ProactiveAudioBench, a benchmark that evaluates the active listening capabilities of audio LLMs. To succeed in the benchmarks, models must speak the right words at the right timings, making it a novel benchmark for interaction. The authors evaluated several large-scale LLMs on the benchmark, showing that they are generally incapable of completing these tasks.

Critical strengths

The authors identify a useful task that SOTA audio LLMs consistently fail on, and release a benchmark to measure these capabilities in future mdoels

Critical weaknesses

It is unclear if the inability of models to perform on this benchmark is due to an architectural limitation, of if they simply have not been trained on it. A section dedicated to performing some fine-tuning on open-source models would have made the paper more complete.

Questions for authors
  1. is there a way you can produce a single score to evaluate models on, like F1, since you already have P and R? This will make it easier to rank.
4 comments
AuthorsOpenReview ↗

Response to Reviewer y7dr

W1: Architectural limitations or insufficient task-specific training?

Thank you for suggesting an investigation of model adaptability. We include an auxiliary sound-event fine-tuning experiment with Qwen3-Omni-30B-A3B-Instruct to examine the effect of task-specific training on notification behavior with the same backbone architecture.

The training set is constructed from 37,976 synthetic audio samples and contains 1,350,802 stepwise dialogue examples, including hard negatives in which the requested target is absent. We use LoRA with rank 8 and alpha 32, a learning rate of 2×10⁻⁶, and a global batch size of 32. Training runs for 10,000 steps on 16 GPUs, and the resulting checkpoint is evaluated.

The evaluation includes 1,112 queries constructed from 500 human-annotated audio samples. The base and fine-tuned models use the same audio, listening instructions, and trigger annotations. The results are shown below:

MetricBase modelAfter fine-tuning
Precision51.21%67.72%
Recall30.52%26.39%
Negative-request FAR15.20%6.05%

Fine-tuning increases Precision and reduces negative-request FAR, while Recall decreases, indicating more conservative notification behavior. These results suggest that task-specific training may help control false alarms, although recall still needs attention. We will include the corresponding experimental settings and results in the manuscript appendix.

AuthorsOpenReview ↗

Response to Reviewer y7dr

Q1: Can F1 provide a single score for model ranking?

Thank you for this suggestion. To facilitate model comparison and ranking, we add macro-averaged joint event F1 for the main-table models, while retaining Recall, Precision, and negative-request FAR to interpret their notification behavior.

Let t denote the task (Detection, Tracking, Counting, or Matching), and d the audio domain (sound, speech, or music). Within each task–domain cell, TP(t,d) counts one-to-one matches satisfying both timing and content requirements, M(t,d) counts required notifications, and K(t,d) counts emitted non-control replies, including incorrect, repeated, and negative-request replies. We define:

F1(t,d) = 2TP(t,d) / [M(t,d) + K(t,d)].

With FP=K−TP counting unmatched replies and FN=M−TP counting unmatched reference notification opportunities, this equals 2TP/(2TP+FP+FN). We first average equally over the three audio domains within each task and then over the four tasks to obtain Overall F1. Timing and content requirements follow the existing matching rules.

The tables below report all 23 model–interface configurations, separately for simulated and native streaming. n denotes scoreable requests; all scores are percentages.

Simulated streaming:

ModelnRecallPrecisionF1FAR
Qwen3.5-Omni-Plus199045.5645.9644.4124.62
Gemini-3.1-Pro199946.9226.4532.8366.42
Gemini-3.8-Flash199334.3722.1826.0863.76
Gemini-3.7-Flash199632.7518.3822.9072.37
Muse-Spark-1.2199923.0320.4017.3959.26
Doubao-Seed-2.0-Lite199917.2313.6313.3582.41
MiniCPM-o 4.5199944.937.6711.7894.78
Qwen2.5-Omni-7B199912.689.658.8532.80
Kimi-Audio-7B199928.882.904.4759.69
Baichuan-Omni-1.5199959.301.873.4699.39
Qwen3-Omni-30B-A3B199916.531.993.1478.32
Voxtral-Small-24B199919.651.753.0173.48
Voxtral-Mini-3B199922.931.542.6599.44
MOSS-Audio-8B199914.501.001.7082.13
Phi-4-Multimodal19991.210.820.702.09
Qwen2-Audio-7B19996.840.210.3990.38
Audio Flamingo Next19990.070.270.100.98

Native streaming:

ModelnRecallPrecisionF1FAR
Qwen3.5-Omni-Plus Realtime199911.8632.2313.7610.32
GPT Realtime 219987.7112.306.8523.05
MiniCPM-o 4.519995.8612.096.6739.93
StepAudio 3 Realtime19994.5415.855.446.11
GPT Live 1†153411.129.527.4737.03
Seeduplex†14834.9714.635.513.12

† Due to reference-audio input limitations in the evaluated interfaces, GPT Live 1 and Seeduplex are evaluated only on Detection, Tracking, and Counting. Their scores average these three tasks; other models average all four.

Under simulated streaming, Qwen3.5-Omni-Plus achieves the highest F1 (44.41%), with relatively balanced Recall and Precision of 45.56% and 45.96%. Baichuan-Omni reaches 59.30% Recall but only 1.87% Precision and 3.46% F1, illustrating that high recall can accompany many unnecessary responses. Joint event F1 accounts for both successful notifications and excess outputs, providing a comparison that does not rely on either metric alone.

Among the four-task native-streaming results, Qwen Realtime achieves the highest F1 at 13.76%, with Recall of 11.86%, indicating that many valid notification opportunities are still missed. StepAudio has a relatively low negative-request FAR of 6.11%, but Recall of 4.54% and F1 of 5.44%, showing that fewer false alarms can also accompany fewer successful notifications. F1 thus provides a summary comparison, while R/P/FAR preserve the distinctions between missed notifications, unnecessary replies, and interruptions on negative requests.

Thank you for the additional results. This would be nice to see within a plot in future versions of the manuscript. I will elect to keep my score.

AuthorsOpenReview ↗

Response to Reviewer y7dr

Thank you for the suggestion and for considering our additional results.

We have added a plot summarizing joint event F1 across the 23 model–interface configurations in the updated manuscript (Figure 9, Appendix D.6), alongside the complete numerical results, to make the comparison easier to follow. The auxiliary fine-tuning setup and the comparison between the base and fine-tuned models are also included in Appendix D.5 (Table 9).

We appreciate your careful review and constructive feedback.

Reviewer_8PAKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduced ProactiveAudioBench, a benchmark that is supposed to evaluate whether audio LMs respond appropriately to events that occur in ongoing audio streams. They studied different types of occurrences, such as first occurrence, every occurrence, etc and evaluated open and closed audio models' performance on a bunch of dimensions such as recall, precision, and false-alarm rate. The problem their benchmark proposes to evaluate seems meaningful, as evaluating proactivity in multimodal systems is conceptually an important direction that the community should be actively thinking about. However, my main concern is that the paper doesn’t provide enough methodological detail to assess the reliability and reproducibility of their benchmark construction. Also, their results are presented in a very shallow manner without useful intuition on model behavior, how the community can build on top of their work, or why their benchmark should even be adopted.

Critical strengths
  • The biggest strength is that the problem they aim to benchmark is conceptually important in itself. The paper addresses proactive, condition-driven responses in audio streams applicable to real-time systems which are becoming more commonplace and preferred by users to interact with models; this evaluation setting is different from conventional audio understanding tasks. Evaluating whether models respond to relevant events at appropriate times, rather than simply whether they can understand audio, is a meaningful research direction.
  • Their evaluation framework is properly validated and improved upon and could be potentially useful, as their metrics seem to capture multiple aspects of proactive model behavior, considering recall, precision, and false-alarm rate. Its utility depends on how reliable the benchmark construction and evaluation procedures are.
Critical weaknesses
  • Benchmarks are generally useful because they provide a standardized way to measure model capabilities and track progress, but this paper insufficiently documents data construction and provides insufficient evidence to assess its reliability. The authors do not adequately explain the provenance of their audio, transcripts, descriptions, and “existing annotations” they claim to use.

  • The authors rely on LLM-assisted annotation procedures but only partially specify which models they used or how they applied them. They name some models (Qwen3-ForcedAligner for speech alignment, PE-A-Frame for boundary refinement, gpt-5.6-luna as the content judge), but others are left unnamed: “the “multimodal models” used to localize sound and music events, the model that assesses background–target compatibility”. Even when they mention a model, they do not explain how they validated the model outputs or whether the outputs were reliable. I am not sure how such a benchmark is meant to be adopted if it is not entirely reproducible.

  • The authors say that human verification was done before they finalized the benchmark instances, but the way they describe it makes it appear less reliable. The annotator instructions and the specific procedures they took to verify the annotations are not described, and it is not even clear who the annotators were.

  • Although the results section has a fair amount of analysis, the paper just reports what models do without critically explaining why. The four tasks in their benchmark require different kinds of skills from each model. Detection requires identifying a target; Tracking requires responding to repeated occurrences; Counting requires respecting a specified occurrence number; Matching requires identifying an event using a reference-audio target. So a bad score could be perception, temporal localization, counting, or memory across the stream, and task-level numbers alone do not tell me which. There is no error analysis whatsoever that could guide future work on proactive audio models?

  • The paper feels slightly incremental in comparison to OmniPro which has a big overlap with some of the categories in the benchmark. Can the authors clarify what new capabilities or failure your task design covers that existing benchmarks do not capture ?

  • I am a bit worried about the ecological validity of their tasks and how they construct the data for them. I am not sure how representative the audio streams are of real continuous audio assistance, or how diverse the conditions really are. Figure 1 shows a few examples and Table 3 in the appendix lists the categories of content they insert, but there is still a huge possibility that some of these examples might be contrived and not reflective of real world use. Also since the authors insert, splice and mix segments in certain cases, I am curious whether the resulting streams still sound natural such that their results and findings are not as a result of artifacts.

Questions for authors

See the weaknesses

8 comments
AuthorsOpenReview ↗

Response to Reviewer 8PAK

W1–W3: Data sources, annotation, and quality validation (Part1/2)

Thank you for your comments on data provenance, annotation models, and human review. We agree that the manuscript needs to explain more clearly how the data were constructed and how annotation and scoring reliability were assessed. We address W1–W3 jointly, covering data sources, model roles, two-stage human quality control, and content-scorer validation.

1. Data sources. ProactiveAudioBench contains 1,999 listening requests: 630 sound, 759 speech, and 610 music requests. Audio sources include YouTube video material and datasets or repositories such as AudioCaps, AudioSet, Freesound/WavCaps, and MTG-Jamendo. We will release the evaluation dataset and a minimal evaluation codebase as soon as possible. The provenance documentation will distinguish source-provided transcripts, descriptions, and event annotations from model-assisted generation or alignment and subsequent human revisions, making their origins traceable.

2. Models and their roles. The “multimodal model” used for sound and music event localization and background–target compatibility assessment is Gemini 3.1 Pro, identified as gemini-3.1-pro-preview in the relevant code. The construction and evaluation components have the following roles:

Model / componentInput and role
Gemini 3.1 ProAssist sound/music event localization; construct question–response pairs using background audio or textual context
Qwen3-ForcedAlignerAlign textual evidence with audio timestamps
PE-A-FrameRefine acoustic event boundaries
gpt-5.6-lunaAssess whether response content satisfies the listening requirement

3. Two-stage human review.

The first stage is review of audio events and temporal annotations. After events and their temporal spans are annotated, two reviewers jointly examine each annotation, checking whether the target event is present and whether its interval agrees with the audio evidence. An annotation is retained only when both approve it.

Verification scopeAccepted / checkedAcceptance rate
Temporal-annotation spot checks161 / 17393.06%

The second stage is review of question–response pairs and trigger annotations. Accepted event records and specific listening conditions are used to construct the pairs and expected response opportunities. Annotators check audio quality, the validity of the pairs, and whether trigger annotations agree with the audio evidence and listening requirements, assigning acceptance, revision, or rejection decisions. Auditors sample batches, and items requiring changes are returned for correction and rechecking. The batch-acceptance target is a sampled pass rate above 98%.

Verification scopeAccepted / checkedAcceptance rate
Initial acceptance checks395 / 44987.97%
Rechecks88 / 8998.88%

The first-stage temporal spot checks achieved 93.06% accuracy; the latest second-stage rechecks achieved a 98.88% pass rate. The revision will include review guidelines and anonymized information on personnel roles, numbers, and backgrounds.

AuthorsOpenReview ↗

Response to Reviewer 8PAK

W1–W3: Data sources, annotation, and quality validation (Part2/2)

4. LLM-judge criteria and human validation. Content scoring uses gpt-5.6-luna, with temperature 0 and a maximum output budget of 2,048 tokens. The evaluation code first identifies temporally eligible reply–reference pairs. The judge then assesses content using the listening instruction, actual reply, reference reply, and textual event evidence, after which the code performs order-preserving one-to-one matching and computes the metrics. Content acceptance requires all of the following:

  1. Actual task completion. The reply must provide the target notification, requested correction, or appropriate tool action.
  2. Correct and relevant content. Correctness and relevance must each score at least 4/5, with no critical factual contradiction affecting task completion. Minor wording issues are acceptable, but substantive errors concerning the target, count, or object of an action cannot pass.
  3. Task-specific requirements. Counting does not require the reply to state N unless the instruction requests it, but any voluntarily reported count must be correct. Matching must identify the reference target, allowing contextually clear references. Tool tasks must pass a programmatic tool-name check, and the judge uses context to assess whether the object and action satisfy the request.

To assess agreement with human judgments, we drew a stratified sample of 100 judging inputs from actual model replies. The sample contains 50 sound and 50 speech examples, with 25 per task and 25 LLM passes and 25 LLM failures within each domain. One reviewer who had not participated in the original annotation assessed the same instructions, replies, references, textual evidence, and rubric. The pass/fail comparison is:

Human passHuman fail
LLM pass491
LLM fail644

The agreement rate is 93%, with Cohen's κ = 0.86, indicating high agreement between the reviewed human judgments and LLM assessments on this sound-and-speech sample. Among the remaining disagreements, six are human passes rejected by the LLM, and one is the reverse. Some disagreements reflect stricter checking of task-specific constraints. For an affordable nearby buffet request, the tested model selected the correct restaurant-search tool but expressed only a generic restaurant search. The human accepted the reply, whereas the judge rejected it for omitting the buffet and price requirements.

We will include the full scoring prompts and decision rules in the appendix and report the human comparison. The data and code will be made public as soon as possible, including model-testing and metric-computation code, accompanied by versioned reproduction instructions.

AuthorsOpenReview ↗

Response to Reviewer 8PAK

W4: Failure analysis and capability diagnostics (Part1/2)

Thank you for noting that task-level scores do not locate the sources of failure. We added factual-question/listening-instruction comparisons, exact counting, all-occurrence localization, and instruction-placement controls to examine audio understanding and conditional notification behavior separately.

1. Correct factual answers can coexist with incorrect notifications. A and B receive the same audio prefix, reference audio, and neutral system prompt. A directly asks about target presence, reference matching, or whether the specified Nth occurrence has happened; B retains the listening instruction. Each task uses 16 positive and 12 negative requests, with each condition independently generating one reply to the provided audio.

The table reports diagnostic accuracy (%). A assesses whether the target-related factual answer is correct; B assesses whether the notification decision and necessary content satisfy the listening instruction and the available audio evidence. Each reply receives a binary correctness judgment. Accuracy is computed within each audio domain and then averaged equally across sound, speech, and music.

ModelDetection A/B (%)Tracking A/B (%)Counting A/B (%)Matching A/B (%)
Qwen3-Omni67.8 / 46.375.0 / 31.742.5 / 15.025.8 / 15.0
MiniCPM-o61.9 / 35.654.2 / 36.745.0 / 50.854.2 / 39.2
Baichuan-Omni53.7 / 29.374.2 / 31.750.8 / 39.247.5 / 15.8
Seed 2.0 Lite82.2 / 50.067.5 / 62.574.2 / 33.368.3 / 60.8

Under the tested configurations, A is higher in 15 of 16 comparisons. In particular, for 11 of 12 Counting negative recordings, Seed correctly states in A that the requested occurrence has not happened but issues an incorrect notification in B. Correct factual judgments therefore do not guarantee compliance with the listening condition.

2. Counting failures involve both counting and notification execution. On the same 28 Counting requests, we compare the original streaming task with two offline question types: whether the requested Nth occurrence has happened, and how many events actually occurred. One actual set of prompts is as follows (translated from Chinese):

  • Original streaming instruction: “When you hear the third triangle strike, immediately notify me.”
  • Threshold question: “Has the third triangle strike already been heard in the main audio?”
  • Exact-count question: “How many triangle strikes occur in the main audio?”

The threshold question asks whether the count has reached the N specified by the original instruction, rather than whether the total is exactly N. Exact counting requires agreement with the actual number of events in the supplied audio. Both offline questions receive the same audio prefix and generate one answer. Each column is averaged over requests within each domain and then equally across sound, speech, and music.

ModelStreaming request success (%)Nth-threshold accuracy (%)Exact-count accuracy (%)
Qwen3-Omni11.742.515.8
MiniCPM-o4.245.021.7
Baichuan-Omni0.050.831.7
Seed 2.0 Lite14.274.256.7

Threshold-question accuracy exceeds exact-count accuracy for all four models. Exact-count accuracy is at most 31.7% for the three open models and 56.7% for Seed. Thus, recovering the precise number of audio events remains difficult even without autonomous notification timing; deciding whether a specified threshold has been reached is comparatively easier. Correct offline judgments also do not guarantee correct streaming execution. In the triangle example, the annotations contain two strikes. Seed answers that the third has not occurred in the threshold probe and reports two strikes in the exact-count probe. Yet its actual streaming trajectory produces “The third triangle strike has now occurred; notifying you” at audio time 3 seconds (translated). This demonstrates an additional difficulty in executing the notification condition.

Reviewer_ZWT5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ProactiveAudioBench, a benchmark for executing listening instructions over continuous audio. It covers sound events, speech, and music through four tasks: Detection requires notification at the first qualifying occurrence; Tracking requires repeated notifications; Counting requires notification only at a specified occurrence; and Matching specifies the target through reference audio. The dataset contains 1,999 requests, including 1,024 positive and 975 negative requests.

The construction pipeline combines model-assisted annotation, controlled audio composition, automated checks, and human review. The evaluation supports simulated and native streaming, using one-to-one response matching to measure joint timing/content recall, precision, and negative-request false alarms.

Critical strengths

I like the setting. The task formulation captures a meaningful gap in audio evaluation, and the evaluation rewards more than frequent correct-sounding responses.

Critical weaknesses

Missing dataset, nor their sources and reproduction details.

A failed inference recorded as waiting can reduce measured recall, make negative-request false-alarm rate appear better, and alter precision and the “No reply” outcome category.

The temporal ground truth and content scorer are insufficiently specified and independently validated.

The timing analysis is greatly shaped by the evaluation protocol.

Questions for authors

See above. I will raise my raise if the author can address my concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZWT5

W1: Data sources and release plans

Thank you for highlighting the shortcomings in data accessibility, provenance, and reproducibility. The submission did not include the dataset or source code. We have compiled a provenance ledger for the final, human-revised set of 1,999 requests, comprising 630 sound, 759 speech, and 610 music requests, with separate records for main audio, reference audio, and inserted materials where applicable.

Examples of sources include YouTube video materials and research datasets or repositories such as AudioCaps, AudioSet, Freesound/WavCaps, and MTG-Jamendo. We will open-source the model-testing and metric-computation code for both simulated and native streaming as soon as possible. We will also release audio and annotations where permitted by their licenses, together with the corresponding versions and reproduction instructions.

W2: Handling inference failures and rerun results

We agree that a technical failure must not be treated as a deliberate waiting decision. Submission deadlines and computational constraints prevented us from completing the verification and reruns of all inference records for the initial submission. We have now completed targeted reruns addressing the issues. The table below presents selected before-and-after results. Most configurations show only small changes in their macro-averaged metrics:

ModelRecall (%)Precision (%)FAR (%)
Phi-41.21 → 1.210.82 → 0.822.19 → 2.09
MiniCPM (offline)44.93 → 44.937.67 → 7.6794.78 → 94.78
Qwen2.5-Omni12.68 → 12.689.65 → 9.6532.80 → 32.80
Qwen3-Omni16.49 → 16.531.11 → 1.1186.97 → 86.97

The table presents only selected results. We will replace the affected results in the updated manuscript PDF with the corrected results that passed these checks.

AuthorsOpenReview ↗

Response to Reviewer ZWT5

W3: Validating temporal annotations and content scoring

Thank you for identifying the need for clearer specification and independent validation of temporal ground truth and content scoring. We clarify temporal annotation and quality control, the content-scoring criteria, and the human comparison below.

Temporal annotation and quality control. Data construction uses two stages of review. In the first stage, Gemini 3.1 Pro assists sound and music event localization, with PE-A-Frame refining event boundaries in applicable subsets; Qwen3-ForcedAligner provides temporal alignment for speech materials. Two reviewers examine each event and temporal annotation, which is retained only when both approve it. Human spot checks of temporal annotations at this stage yielded the following results:

Verification scopeAccurate / checkedAccuracy
Temporal annotation checks161 / 17393.06%

In the second stage, the approved data are used to construct question–answer pairs. Audio quality, question–answer validity, and notification timing are then checked, followed by category-based spot checks, revision, and rechecking. Historical acceptance results are shown below:

Verification scopeAccepted / checkedAcceptance rate
Initial acceptance checks395 / 44987.97%
Rechecks after revision88 / 8998.88%

The acceptance rate in rechecks after revision reached 98.88%.

Content-scoring criteria. The LLM judge is gpt-5.6-luna, with temperature 0 and a maximum output budget of 2,048 tokens. It receives the listening instruction, actual reply, reference reply, and textual event evidence to determine whether the reply fulfills the current listening task. Content acceptance requires all of the following:

  1. Actual task completion. The reply must provide the target notification, requested correction, or appropriate tool action. For example, “A dog barked” can fulfill a notification request, whereas “I will notify you when a dog barks” is only a promise.
  2. Correct and relevant content. Correctness and relevance must each score at least 4/5, with no critical factual contradiction affecting task completion. A correctness score of 5 indicates that the core requirement is met without known substantive errors; 4 permits minor wording issues that do not alter the facts. Substantive errors concerning the target, occurrence count, or object of an action cannot pass. Conciseness is recorded separately and does not determine acceptance.
  3. Task-specific requirements. When the instruction does not explicitly request a count in the reply, Counting does not require the reply to state N, but any voluntarily reported count must be correct. Matching must identify the reference target, allowing contextually clear references. Tool tasks must pass a programmatic tool-name check, and the judge uses context to assess whether the object and action satisfy the request.

Human comparison. We sampled 100 real judging inputs: 50 sound and 50 speech examples, with 25 examples per task. One reviewer who had not participated in the original annotation scored them using the same textual evidence and rules. The authors subsequently reviewed disagreements with access to the judge's rationales. The pass/fail judgments after review are shown below:

Human passHuman fail
LLM pass491
LLM fail644

After review, 93 of the 100 judgments agree, yielding 93% agreement and Cohen’s κ = 0.86.

The reviewed human judgments and LLM assessments show high agreement on this sound-and-speech sample. Some disagreements also illustrate the judge's stricter checking of specific task constraints. For example, for an affordable nearby buffet request, the model selected the correct restaurant-search tool but expressed only a generic restaurant search. The human accepted the reply, whereas the judge found that it did not sufficiently address the buffet and price requirements. This example shows that the judge checks whether the reply covers the request's key constraints in addition to selecting the correct tool category. We will include the complete scoring prompts in the appendix for further inspection.

AuthorsOpenReview ↗

Response to Reviewer ZWT5

W4: Evaluation protocols and temporal windows

We thank the reviewer for highlighting the relationship between timing analysis and the evaluation protocol. We agree that the distributions must be interpreted together with how replies are elicited and temporally matched. Reference timestamps are intended to mark the earliest notification opportunity at which the listening condition is satisfied and sufficient audio evidence is available, rather than uniformly marking event onsets.

To quantify the effect of temporal tolerance, we report the same nine simulated-streaming and four native-streaming configurations as in the main table, comparing checkpoint-offset and continuous-time windows, respectively. For simulated streaming, GT is mapped to reference checkpoint i using the original rule, and reply checkpoint j is eligible when a≤j−i≤b. Thus, ±1 checkpoint and [−1,+1] checkpoints are identical and share the default column. Native streaming instead checks whether the continuous-time difference between the reply and GT falls within [a,b] seconds. The tables retain equal averaging over the three domains and four tasks.

Simulated streaming (checkpoint offsets; each cell gives Recall/Precision, %):

Model[−1,+1] (default)[−1,+2][−1,+3][−1,+4][−1,+5][−2,+2][−3,+3]
Baichuan-Omni-1.559.30/1.8761.23/1.9462.13/1.9762.32/1.9962.35/1.9962.62/1.9864.11/2.03
Qwen3-Omni-30B-A3B16.53/1.9919.48/2.1922.02/2.3222.31/2.3722.45/2.3821.15/2.3724.81/2.63
Kimi-Audio-7B28.88/2.9029.76/2.9930.09/3.0230.32/3.0530.36/3.0630.65/3.0531.33/3.11
MiniCPM-o 4.544.93/7.6747.30/8.1348.41/8.3948.60/8.4448.64/8.4450.96/8.8854.29/9.50
Gemini-3.1-Pro46.92/26.4548.50/27.2349.59/27.7149.94/27.8650.16/27.9552.30/29.0055.88/30.77
Gemini-3.7-Flash32.75/18.3834.76/19.4736.66/20.5036.92/20.6736.96/20.6938.17/21.1342.69/23.54
Qwen3.5-Omni-Plus45.56/45.9645.95/46.3646.14/46.5546.26/46.6446.35/46.7450.31/49.8652.13/51.44
Muse-Spark-1.223.03/20.4026.24/23.5028.02/24.4828.67/24.9728.83/25.0328.54/25.3031.32/26.86
Doubao-Seed-2.0-Lite17.23/13.6318.00/14.2018.83/14.8518.93/14.9018.93/14.9021.90/16.3526.53/19.04

Native streaming (offsets in seconds; each cell gives Recall/Precision, %):

Model[−1,+1][−1,+2][−1,+3] (default)[−1,+4][−1,+5][−2,+2][−3,+3]
MiniCPM-o 4.51.98/4.364.39/9.055.86/12.097.36/14.378.54/16.844.50/9.285.98/12.32
Qwen3.5-Omni-Plus Realtime4.10/9.377.62/16.6811.86/32.2315.17/39.4217.41/43.007.71/16.8512.17/32.71
GPT Realtime 21.40/3.324.52/8.817.71/12.3011.55/17.0814.00/19.534.90/9.788.43/13.66
StepAudio 3 Realtime2.16/9.273.27/11.894.54/15.855.24/17.385.93/18.603.61/12.575.06/16.72

Simulated streaming provides an explicit decision opportunity at every checkpoint. On the observed traces, extending the late boundary yields limited gains for some configurations: expanding from [−1,+1] to [−1,+3] checkpoints increases Recall from 45.56% to 46.14% for Qwen3.5-Omni-Plus and from 46.92% to 49.59% for Gemini-3.1-Pro. Because wider windows also credit notifications several decision steps away from the reference, we retain ±1 checkpoint as the default and report wider-window sensitivity results.

Native streaming shows a clear benefit from additional late tolerance: all four configurations achieve higher Recall under [−1,+3] s than under the equally wide [−2,+2] s window. Extending the late boundary from +2 s to +3 s increases Qwen3.5-Omni-Plus Realtime's Recall from 7.62% to 11.86% and Precision from 16.68% to 32.23%; GPT Realtime 2's Recall also rises from 4.52% to 7.71%. One possible contributor is the response-triggering mechanism: recorded session acknowledgments confirm that server VAD was enabled in our GPT Realtime 2 and Qwen3.5-Omni-Plus Realtime runs. Even when the listening condition is satisfied within a speech segment, the interface may wait until it detects the end of speech before initiating a reply, delaying the response relative to the reference timestamp. We therefore use [−1,+3] s to balance allowance for short response-initiation waits against the need to penalize late notifications.

AuthorsOpenReview ↗

Response to Reviewer ZWT5

Thank you again for your careful review. We have updated the manuscript to incorporate the clarifications and additional results from our response, with the revisions marked in blue:

  • Data sources and annotation details: Section 3.3 and Appendix D.7 now describe the sources, construction tools, and annotation procedures. The Reproducibility Statement also specifies our data and code release plans.
  • Inference failures and corrected results: Appendix B clarifies the treatment of technical failures and unresolved records in the main evaluation, and Tables 2 and 4 report the updated results.
  • Temporal annotations and content scoring: Appendices D.1, D.2, and D.4 provide the review procedures, human validation results, scoring criteria, and simplified judging prompt.
  • Protocol-dependent timing analysis: Section 5.3 and Appendix D.3 clarify the interpretation of timing distributions and provide the window-sensitivity analysis and response-triggering settings.

If you find that these revisions address the concerns you raised, we would be grateful if you could reconsider your assessment and the corresponding score. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.