acceptodds
Under review as a conference paper at ICLR 2027

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

Abstract

Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBENCH, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VOXGATE as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.

Primary area:
datasets and benchmarks
Keywords:
Audio Large Language Models, Voice Assistants, Agentic AI, Paralinguistic Understanding, Tool Use Safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QR2KOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VGBench, a 1,018-item diagnostic benchmark that separates command recognition from acoustic-context gating in audio-LLM voice agents. Every item maps to a shared action space (Mute / Tool / Answer). The benchmark has three families: side-talk (395 real recordings; one speaker produces two consecutive utterances, one addressed to the assistant and one to a nearby person), self-talk (223 real recordings; command-like phrases embedded in seven non-instructional discourse frames, all targeting Mute), and speaker-switch (400 TTS-synthesized counterfactual pairs; the same context+trigger text is rendered as same-speaker/near-field → Tool, or as a different-voice/far-field trigger after a 600 ms gap → Mute, with a text-only control → Tool). Six raw audio LLMs and three training-free adaptations select the target tool at high rates on the positive controls but mute at most 14% of switched commands. The authors then post-train Qwen3-Omni-30B (VoxGate: LoRA SFT on the VGBench training split mixed with WearVox data, followed by an exploratory pair-grouped GRPO stage), raising held-out switch muting from 0% to 91.3–92.5% while keeping positive controls at 100% and preserving WearVox downstream performance. A factorized cue study and a cue-balanced SFT control show that the learned gate responds to source change independently, but that its sensitivity to far-field rendering is scene-dependent.

Critical strengths
  • The action-level framing is clear and practically relevant. Measuring whether acoustic and conversational context changes an executable decision (mute vs. tool call), rather than a perceptual label or a free-form response, matches how voice agents are deployed, and pairing every mute score with same-speaker and text-only tool-selection controls, with no aggregate score, prevents a trivial always-mute policy from looking good.

  • The side-talk and self-talk families are axes that existing interruption and bystander benchmarks do not cover. Holding speaker identity fixed to test recipient attribution, and embedding command cores in discourse frames (planning, regret, quotation, sarcasm, etc.), isolate addressedness failures that are independent of speaker change, and both families use real recordings.

  • The factorized cue study (Tables 3/11) and the cue-balanced SFT control (Tables 12/13) are a useful piece of self-diagnosis: they decompose the learned gate into source, distance, and boundary effects and expose that the distance effect is tied to a rendering discontinuity. The limitations section is candid about the single-run, single-split, item-level-only evaluation.

Critical weaknesses
  • Missing comparison with closely related prior work. The paper’s central device — holding the transcript fixed while changing only the speaker acoustics, then training on those counterfactual pairs to suppress text-based shortcuts — is the design of TPI-Bench’s Janus-Test and speaker-aware hard negatives (e.g. [1][2]). The shortcut finding, the retention check on a general VA benchmark, and the negative result for prompting also appear there. I recognize that the scenario differs (the bystander issues the command; the output is a tool call), but the design philosophy and the message are the same, and the paper does not cite this work; Related Work instead names VoxParadox as the closest design. Moreover, [1] covers substantially more: an actionable/ignorable policy, 88K training and 4K diverse test items, a real-world test set, human evaluation, and significance tests. Authors need to clarify the omission of citations and what difference should readers learn from the manuscript .

  • Headline result is synthetic and partly an artifact. All switch audio is TTS, with no real bystander recordings and no real-world transfer test. The switch condition changes source, distance, and a 600 ms gap simultaneously, and the paper’s own Table 13 shows that with scene-consistent rendering, different/far/600 muting drops from 91.25% to 46.25% and same/far from 60% to 6.25% — roughly half of the 91.3% headline reflects a context–trigger rendering discontinuity rather than distance. The split is item-level only (voices, templates, and tools are shared), and post-SFT self-talk (52%) and side-talk (68%) suggest the learned gate is specific to the switch distribution. The learnability claim is therefore supported only within the benchmark’s own synthetic data.

  • Limited evaluation rigor and missing baselines. Results come from one run and one split with 80 held-out switch commands, so the GRPO gain (73→74) is a single item and no significance tests are reported. There is no cascade baseline (speaker verification + ASR + LLM), so the need for end-to-end gating is not shown. The side-talk judge has no reported human agreement, the self-talk family has no positive control (an always-mute policy scores 100%), and the audio corpus is unavailable at review time.

[1] Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, & Sungroh Yoon. (2026). Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions. [2] Xiao Zhan, Guangzhi Sun, Jose Such, & Phil Woodland. (2026). Protecting Bystander Privacy via Selective Hearing in Audio LLMs.

Questions for authors

See weakness

3 comments

Q1. Missing comparison with closely related prior work (TPI-Bench Janus-Test, [1], [2])

The central device, holding the transcript fixed while changing only speaker acoustics and training on counterfactual pairs to suppress text-based shortcuts, is the design of TPI-Bench's Janus-Test and speaker-aware hard negatives (e.g., [1][2]). The shortcut finding, the retention check on a general VA benchmark, and the negative result for prompting also appear there. Although the scenario differs (the bystander issues the command, the output is a tool call), the design philosophy and message are the same, yet the paper does not cite this work; Related Work instead names VoxParadox as the closest design. Moreover, [1] covers substantially more: an actionable/ignorable policy, 88K training and 4K diverse test items, a real-world test set, human evaluation, and significance tests. The authors need to clarify the omission of citations and what difference should readers learn from this manuscript.

References cited by the reviewer:

  • [1] Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, & Sungroh Yoon. (2026). Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions.
  • [2] Xiao Zhan, Guangzhi Sun, Jose Such, & Phil Woodland. (2026). Protecting Bystander Privacy via Selective Hearing in Audio LLMs.

A1. We thank the reviewer for pointing us to this work, which we should have cited. We agree that the high-level design philosophy overlaps with Janus-Test and with [1] and [2]: holding the transcript fixed, changing speaker acoustics, training on counterfactual pairs, and observing that prompting alone does not fix the problem. We will add [1], [2], and TPI-Bench to Related Work and position VGBench against them explicitly rather than naming VoxParadox as the only closest design.

The differences we want readers to take away are the following. First, our output is an executable action in a shared space of Mute, Tool, and Answer, so we measure whether acoustic context changes a decision to act, not a perceptual label or a free-form response. Second, every mute score is paired with same-speaker and text-only tool-selection controls, so an always-mute policy cannot appear strong. Third, the benchmark spans three addressedness families, two of which use real recordings. Fourth, the factorized cue study (Tables 3 and 11) and the cue-balanced SFT control (Tables 12 and 13) decompose the learned gate into source, distance, and boundary effects, which is the diagnostic contribution of this manuscript. We will add a short comparison that makes these distinctions explicit and acknowledge the larger scale and real-world evaluation of [1].

Q2. Headline result is synthetic and partly an artifact

All switch audio is TTS, with no real bystander recordings and no real-world transfer test. The switch condition changes source, distance, and a 600 ms gap simultaneously, and the paper's own Table 13 shows that with scene-consistent rendering, different/far/600 muting drops from 91.25% to 46.25% and same/far from 60% to 6.25%, meaning roughly half of the 91.3% headline reflects a context-trigger rendering discontinuity rather than distance. The split is item-level only (voices, templates, and tools are shared), and post-SFT self-talk (52%) and side-talk (68%) suggest the learned gate is specific to the switch distribution. The learnability claim is therefore supported only within the benchmark's own synthetic data.

A2. We thank the reviewer for this detailed assessment. We agree with most of it, and much of it is already our own analysis. Table 13 reports exactly the scene-consistent rendering numbers the reviewer cites, and we state in the paper that the distance effect is tied to a rendering discontinuity. We do not claim distance-invariant speaker verification, nor do we claim real-world transfer. Our learnability claim is narrower: the post-trained gate responds to the speaker-source change independently of the other cues, while its distance and boundary sensitivity is scene-dependent.

To keep the headline from being misread, we will revise the text to foreground the artifact analysis and to present the 91.25% figure together with the scene-consistent 46.25% rather than in isolation. We also added full-corpus evaluation on newer models, where qwen3.5-omni-flash mutes 8.0% of switched commands and gpt-audio-1.5 mutes 0.0%, confirming that the failure is not tied to one model. We acknowledge that the split is item-level and that the post-SFT self-talk and side-talk numbers show the gate is specific to the switch distribution, and we will scope the conclusion to the benchmark accordingly.

Q3. Limited evaluation rigor and missing baselines

Results come from one run and one split with 80 held-out switch commands, so the GRPO gain (73→74) is a single item and no significance tests are reported. There is no cascade baseline (speaker verification + ASR + LLM), so the need for end-to-end gating is not shown. The side-talk judge has no reported human agreement, the self-talk family has no positive control (an always-mute policy scores 100%), and the audio corpus is unavailable at review time.

A3. We thank the reviewer for these observations and take them as clear limitations of the present study. We agree that the results come from one run and one split with 80 held-out switch commands, and that the GRPO change from 73 to 74 is a single item. We do not draw a strong claim from this difference and describe the GRPO stage as exploratory in the paper. Adding further runs, significance tests, a second split, a cascade baseline, human agreement for the side-talk judge, a self-talk positive control, and a public audio release would all strengthen the evaluation, and we agree each is worthwhile. These are beyond the scope of the current version, and we have framed our contribution accordingly as a controlled diagnostic of the native action policy rather than a fully validated deployment system. We will state these limitations explicitly in the revision instead of overclaiming, and we will clarify in the discussion that dedicated speaker verification, diarization, and DSP front ends are complementary system-level mechanisms to the native gating studied here.

Reviewer_L3kjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose a benchmark called VGBench for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Side-talk contains 396 recording, self-talk contains 223 recordings, and speaker-switch contains 400 speaker-switch items by TTS and RIR+room rendering. The experiments on six audio LLMs and three training-free adaptations has poor performance on speaker-switch tasks, i.e., highest raw switch mute rate is 14%. The paper use 80% VGBench in-domain data and WearVox through SFT and GRPO, and it improves the performance on three tasks.

Critical strengths
  1. Side-talk, self-talk, and speaker-switching represent real-world challenges for conversational models and are topics worthy of further research. This paper establishes a practical and valuable benchmark within a controlled experimental environment.
  2. The data for side-talk and self-talk consist of actual recordings, making them more representative of real-world scenarios.
  3. The paper provides a comprehensive comparison of various models, as well as an evaluation of different training-free adaptation methods.
Critical weaknesses
  1. The most challenging task, "speaker-switch," is based on simulated data rather than actual recordings; consequently, the distribution of this data differs from that of real-world scenarios.
  2. Improvements in the "speaker-switch" task achieved through the use of in-domain data do not necessarily transfer to improvements in real-world scenarios.
  3. There is a lack of evaluation for this task using full-duplex models.
Questions for authors

Why was the use of recorded audio not considered for the speaker-switch task?

1 comment

We thank the reviewer for this question.

We did consider recorded audio, but prioritized scalability and experimental control. Recording hundreds of tightly matched counterfactual pairs (identical transcript and tool target, same speaker versus different speaker, near versus far, with and without the gap) is impractical at the scale we needed. Synthesis let us generate exactly matched pairs and scale to 400 switch items with clean causal contrasts. Since the synthetic construction produced a diagnostic signal that also appears in newer models, we chose it for this family, while keeping side-talk and self-talk as real recordings.

Reviewer_EPDVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether audio language models use conversational and acoustic context to decide whether to remain silent, answer, or select a tool. It introduces VGBENCH. Each speaker-switch case includes same-speaker, switched-speaker, and text-only conditions. The specified words remain fixed, while the main acoustic contrast jointly changes the trigger’s speaker, distance rendering, and temporal boundary.

Evaluation finds low switch muting despite much stronger tool selection on positive controls. The paper then presents VOXGATE, a Qwen-based post-training case study. On 80 held-out switch cases, supervised training correctly mutes 73 while retaining correct tool-name selection on all same-speaker and text-only controls. An exploratory GRPO stage changes this to 74 and reports additional side-talk and self-talk improvements.

The principal contributions are the controlled action contrasts, the behavioral diagnosis, and the analysis of which cues support the learned behavior.

Critical strengths

I like the setup of the benchmark. The concept of speaker-switch is an interesting one, and it's a task that requires the model's native audio input capability to solve it.

Critical weaknesses

The category classification is unclear and unfair. For example, sidetalk ST_ai_first_fc_fc_ds01__20260903_155946.wav doesn't sounds like a sidetalk at all; from a human judgement, the loudness and the way you speak makes it like the same person asking two questions at once (create a workout playlist, and asking how old is michael jackson), instead of a core action and then a "side-talk". I think if that's what we are penalizing the agent for, that's not fair. Or as another example, selftalk A_venting_F12_joking_threat_cmd_set_reminder__20260903_154434.wav is even hard for a human to tell if a action (add to remind list) should be made. It's a subjective decision, and we shouldn't use that kind of question to score the model. Because of the quality of the dataset sample I got, I doubt the quality of the benchmark and believe the conclusion you get from it is also not trustworthy.

6 models are small for a benchmark paper, and some of the models are pretty outdated (like qwen 3 omni). I want to see if this trend are still there for these newer models. For example, gpt-realtime, qwen3.5-omni-plus, grok-voice, etc.

The benchmark counts every non-[Mute] output as a false trigger, but the main switch results report only mute rates. They do not separate tool calls, natural-language refusals, other answers, and malformed outputs.

The validation leaves substantial uncertainty about what generalizes beyond the synthetic source-and-scene construction.

Questions for authors

See weakness

1 comment

Q1. Category classification is unclear and unfair, casting doubt on dataset and conclusion quality

The category labels are claimed to be ambiguous and unfair. Two examples are given:

  • sidetalk ST_ai_first_fc_fc_ds01__20260903_155946.wav does not sound like sidetalk. In human judgment, the loudness and speaking style make it sound like one person asking two questions at once (create a workout playlist, and ask how old Michael Jackson is), rather than a core action followed by a side-talk. Penalizing the agent for this is unfair.
  • selftalk A_venting_F12_joking_threat_cmd_set_reminder__20260903_154434.wav is even hard for a human to decide whether an action (add to reminder list) should be taken. This is a subjective decision and should not be used to score the model. Because of the sampled data quality, the reviewer doubts the benchmark quality and finds the conclusions untrustworthy.

A1. We thank the reviewer for checking individual samples so carefully. We agree that this kind of speech is genuinely difficult to synthesize, and that the boundary between one speaker asking two questions and a genuine side-talk can be subtle. During recording we asked the speakers to deliver the bystander turn with different vocal effort and spatial placement, so the two turns carry an acoustic flow and direction change rather than only a text difference. We invite the reviewer to listen again to ST_ai_first_fc_fc_ds01__20260903_155946.wav: the second utterance arrives from a different direction and distance than the first, which is the cue the item is designed to test. For the selftalk example, we acknowledge that the joking-threat frame makes the intended action genuinely ambiguous, and we agree that this affects a small number of items. We will make the annotation criteria explicit in the revision so that the labeling rule is clear; we do not claim that every borderline item has a unique human label.

Q2. Only six models, some outdated; trend should be checked on newer models

Six models are considered too few for a benchmark paper, and some are outdated (e.g., Qwen3-Omni). The reviewer wants to see whether the same trend holds for newer models such as gpt-realtime, qwen3.5-omni-plus, and grok-voice.

A2. We agree that the model list should include more recent systems. We have added full-corpus speaker-switch results for two recent models, qwen3.5-omni-flash and gpt-audio-1.5, evaluated with the same 400 switch items and three conditions at temperature zero. qwen3.5-omni-flash mutes 8.0% of switched commands (32/400 under the broadened [Mute] parser) while selecting the target tool on 90.5% of them; gpt-audio-1.5 mutes 0.0% while selecting the target tool on 99.5%. The low switch-muting behavior therefore persists in newer models, which strengthens the diagnostic finding. For gpt-audio-1.5 the text-only condition is not applicable because the model requires audio input and returned API errors on all 400 text-only items; we report this explicitly rather than imputing a value. We will add these results to the main table in the revision.

Q3. Non-[Mute] outputs are all counted as false triggers, but only mute rates are reported

The benchmark counts every non-[Mute] output as a false trigger, yet the main switch results report only mute rates. Tool calls, natural-language refusals, other answers, and malformed outputs are not separated.

A3. This is a fair point, and we have re-tabulated the predictions by execution consequence rather than by the binary mute/non-mute split. We now separate every output into six categories: [Mute], target tool, wrong tool, natural-language refusal, other answer, and parse failure, for the switch condition and both positive controls. On qwen3.5-omni-flash, 363 of the 368 non-mute switch outputs are genuine tool calls, with 1 wrong tool, 0 refusals, 1 clarifying answer, and 1 parse failure. On gpt-audio-1.5, 399 of 400 non-mute switch outputs are tool calls, with 1 wrong tool and 0 refusals. In other words, nearly all non-mute outputs are real tool executions, so the binary count is conservative and does not inflate the failure rate with harmless refusals. We will report the full six-way breakdown in the revision and use "dangerous misexecution" only for the tool-call categories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.