- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ProactiveAudioBench, a benchmark that evaluates the active listening capabilities of audio LLMs. To succeed in the benchmarks, models must speak the right words at the right timings, making it a novel benchmark for interaction. The authors evaluated several large-scale LLMs on the benchmark, showing that they are generally incapable of completing these tasks.
Critical strengths
The authors identify a useful task that SOTA audio LLMs consistently fail on, and release a benchmark to measure these capabilities in future mdoels
Critical weaknesses
It is unclear if the inability of models to perform on this benchmark is due to an architectural limitation, of if they simply have not been trained on it. A section dedicated to performing some fine-tuning on open-source models would have made the paper more complete.
Questions for authors
- is there a way you can produce a single score to evaluate models on, like F1, since you already have P and R? This will make it easier to rank.
4 comments
Response to Reviewer y7dr
Q1: Can F1 provide a single score for model ranking?
Thank you for this suggestion. To facilitate model comparison and ranking, we add macro-averaged joint event F1 for the main-table models, while retaining Recall, Precision, and negative-request FAR to interpret their notification behavior.
Let t denote the task (Detection, Tracking, Counting, or Matching), and d the audio domain (sound, speech, or music). Within each task–domain cell, TP(t,d) counts one-to-one matches satisfying both timing and content requirements, M(t,d) counts required notifications, and K(t,d) counts emitted non-control replies, including incorrect, repeated, and negative-request replies. We define:
F1(t,d) = 2TP(t,d) / [M(t,d) + K(t,d)].
With FP=K−TP counting unmatched replies and FN=M−TP counting unmatched reference notification opportunities, this equals 2TP/(2TP+FP+FN). We first average equally over the three audio domains within each task and then over the four tasks to obtain Overall F1. Timing and content requirements follow the existing matching rules.
The tables below report all 23 model–interface configurations, separately for simulated and native streaming. n denotes scoreable requests; all scores are percentages.
Simulated streaming:
| Model | n | Recall | Precision | F1 | FAR |
|---|---|---|---|---|---|
| Qwen3.5-Omni-Plus | 1990 | 45.56 | 45.96 | 44.41 | 24.62 |
| Gemini-3.1-Pro | 1999 | 46.92 | 26.45 | 32.83 | 66.42 |
| Gemini-3.8-Flash | 1993 | 34.37 | 22.18 | 26.08 | 63.76 |
| Gemini-3.7-Flash | 1996 | 32.75 | 18.38 | 22.90 | 72.37 |
| Muse-Spark-1.2 | 1999 | 23.03 | 20.40 | 17.39 | 59.26 |
| Doubao-Seed-2.0-Lite | 1999 | 17.23 | 13.63 | 13.35 | 82.41 |
| MiniCPM-o 4.5 | 1999 | 44.93 | 7.67 | 11.78 | 94.78 |
| Qwen2.5-Omni-7B | 1999 | 12.68 | 9.65 | 8.85 | 32.80 |
| Kimi-Audio-7B | 1999 | 28.88 | 2.90 | 4.47 | 59.69 |
| Baichuan-Omni-1.5 | 1999 | 59.30 | 1.87 | 3.46 | 99.39 |
| Qwen3-Omni-30B-A3B | 1999 | 16.53 | 1.99 | 3.14 | 78.32 |
| Voxtral-Small-24B | 1999 | 19.65 | 1.75 | 3.01 | 73.48 |
| Voxtral-Mini-3B | 1999 | 22.93 | 1.54 | 2.65 | 99.44 |
| MOSS-Audio-8B | 1999 | 14.50 | 1.00 | 1.70 | 82.13 |
| Phi-4-Multimodal | 1999 | 1.21 | 0.82 | 0.70 | 2.09 |
| Qwen2-Audio-7B | 1999 | 6.84 | 0.21 | 0.39 | 90.38 |
| Audio Flamingo Next | 1999 | 0.07 | 0.27 | 0.10 | 0.98 |
Native streaming:
| Model | n | Recall | Precision | F1 | FAR |
|---|---|---|---|---|---|
| Qwen3.5-Omni-Plus Realtime | 1999 | 11.86 | 32.23 | 13.76 | 10.32 |
| GPT Realtime 2 | 1998 | 7.71 | 12.30 | 6.85 | 23.05 |
| MiniCPM-o 4.5 | 1999 | 5.86 | 12.09 | 6.67 | 39.93 |
| StepAudio 3 Realtime | 1999 | 4.54 | 15.85 | 5.44 | 6.11 |
| GPT Live 1† | 1534 | 11.12 | 9.52 | 7.47 | 37.03 |
| Seeduplex† | 1483 | 4.97 | 14.63 | 5.51 | 3.12 |
† Due to reference-audio input limitations in the evaluated interfaces, GPT Live 1 and Seeduplex are evaluated only on Detection, Tracking, and Counting. Their scores average these three tasks; other models average all four.
Under simulated streaming, Qwen3.5-Omni-Plus achieves the highest F1 (44.41%), with relatively balanced Recall and Precision of 45.56% and 45.96%. Baichuan-Omni reaches 59.30% Recall but only 1.87% Precision and 3.46% F1, illustrating that high recall can accompany many unnecessary responses. Joint event F1 accounts for both successful notifications and excess outputs, providing a comparison that does not rely on either metric alone.
Among the four-task native-streaming results, Qwen Realtime achieves the highest F1 at 13.76%, with Recall of 11.86%, indicating that many valid notification opportunities are still missed. StepAudio has a relatively low negative-request FAR of 6.11%, but Recall of 4.54% and F1 of 5.44%, showing that fewer false alarms can also accompany fewer successful notifications. F1 thus provides a summary comparison, while R/P/FAR preserve the distinctions between missed notifications, unnecessary replies, and interruptions on negative requests.
Thank you for the additional results. This would be nice to see within a plot in future versions of the manuscript. I will elect to keep my score.
Response to Reviewer y7dr
Thank you for the suggestion and for considering our additional results.
We have added a plot summarizing joint event F1 across the 23 model–interface configurations in the updated manuscript (Figure 9, Appendix D.6), alongside the complete numerical results, to make the comparison easier to follow. The auxiliary fine-tuning setup and the comparison between the base and fine-tuned models are also included in Appendix D.5 (Table 9).
We appreciate your careful review and constructive feedback.
Response to Reviewer y7dr
W1: Architectural limitations or insufficient task-specific training?
Thank you for suggesting an investigation of model adaptability. We include an auxiliary sound-event fine-tuning experiment with Qwen3-Omni-30B-A3B-Instruct to examine the effect of task-specific training on notification behavior with the same backbone architecture.
The training set is constructed from 37,976 synthetic audio samples and contains 1,350,802 stepwise dialogue examples, including hard negatives in which the requested target is absent. We use LoRA with rank 8 and alpha 32, a learning rate of 2×10⁻⁶, and a global batch size of 32. Training runs for 10,000 steps on 16 GPUs, and the resulting checkpoint is evaluated.
The evaluation includes 1,112 queries constructed from 500 human-annotated audio samples. The base and fine-tuned models use the same audio, listening instructions, and trigger annotations. The results are shown below:
Fine-tuning increases Precision and reduces negative-request FAR, while Recall decreases, indicating more conservative notification behavior. These results suggest that task-specific training may help control false alarms, although recall still needs attention. We will include the corresponding experimental settings and results in the manuscript appendix.