- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VGBench, a 1,018-item diagnostic benchmark that separates command recognition from acoustic-context gating in audio-LLM voice agents. Every item maps to a shared action space (Mute / Tool / Answer). The benchmark has three families: side-talk (395 real recordings; one speaker produces two consecutive utterances, one addressed to the assistant and one to a nearby person), self-talk (223 real recordings; command-like phrases embedded in seven non-instructional discourse frames, all targeting Mute), and speaker-switch (400 TTS-synthesized counterfactual pairs; the same context+trigger text is rendered as same-speaker/near-field → Tool, or as a different-voice/far-field trigger after a 600 ms gap → Mute, with a text-only control → Tool). Six raw audio LLMs and three training-free adaptations select the target tool at high rates on the positive controls but mute at most 14% of switched commands. The authors then post-train Qwen3-Omni-30B (VoxGate: LoRA SFT on the VGBench training split mixed with WearVox data, followed by an exploratory pair-grouped GRPO stage), raising held-out switch muting from 0% to 91.3–92.5% while keeping positive controls at 100% and preserving WearVox downstream performance. A factorized cue study and a cue-balanced SFT control show that the learned gate responds to source change independently, but that its sensitivity to far-field rendering is scene-dependent.
Critical strengths
-
The action-level framing is clear and practically relevant. Measuring whether acoustic and conversational context changes an executable decision (mute vs. tool call), rather than a perceptual label or a free-form response, matches how voice agents are deployed, and pairing every mute score with same-speaker and text-only tool-selection controls, with no aggregate score, prevents a trivial always-mute policy from looking good.
-
The side-talk and self-talk families are axes that existing interruption and bystander benchmarks do not cover. Holding speaker identity fixed to test recipient attribution, and embedding command cores in discourse frames (planning, regret, quotation, sarcasm, etc.), isolate addressedness failures that are independent of speaker change, and both families use real recordings.
-
The factorized cue study (Tables 3/11) and the cue-balanced SFT control (Tables 12/13) are a useful piece of self-diagnosis: they decompose the learned gate into source, distance, and boundary effects and expose that the distance effect is tied to a rendering discontinuity. The limitations section is candid about the single-run, single-split, item-level-only evaluation.
Critical weaknesses
-
Missing comparison with closely related prior work. The paper’s central device — holding the transcript fixed while changing only the speaker acoustics, then training on those counterfactual pairs to suppress text-based shortcuts — is the design of TPI-Bench’s Janus-Test and speaker-aware hard negatives (e.g. [1][2]). The shortcut finding, the retention check on a general VA benchmark, and the negative result for prompting also appear there. I recognize that the scenario differs (the bystander issues the command; the output is a tool call), but the design philosophy and the message are the same, and the paper does not cite this work; Related Work instead names VoxParadox as the closest design. Moreover, [1] covers substantially more: an actionable/ignorable policy, 88K training and 4K diverse test items, a real-world test set, human evaluation, and significance tests. Authors need to clarify the omission of citations and what difference should readers learn from the manuscript .
-
Headline result is synthetic and partly an artifact. All switch audio is TTS, with no real bystander recordings and no real-world transfer test. The switch condition changes source, distance, and a 600 ms gap simultaneously, and the paper’s own Table 13 shows that with scene-consistent rendering, different/far/600 muting drops from 91.25% to 46.25% and same/far from 60% to 6.25% — roughly half of the 91.3% headline reflects a context–trigger rendering discontinuity rather than distance. The split is item-level only (voices, templates, and tools are shared), and post-SFT self-talk (52%) and side-talk (68%) suggest the learned gate is specific to the switch distribution. The learnability claim is therefore supported only within the benchmark’s own synthetic data.
-
Limited evaluation rigor and missing baselines. Results come from one run and one split with 80 held-out switch commands, so the GRPO gain (73→74) is a single item and no significance tests are reported. There is no cascade baseline (speaker verification + ASR + LLM), so the need for end-to-end gating is not shown. The side-talk judge has no reported human agreement, the self-talk family has no positive control (an always-mute policy scores 100%), and the audio corpus is unavailable at review time.
[1] Dongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim, & Sungroh Yoon. (2026). Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions. [2] Xiao Zhan, Guangzhi Sun, Jose Such, & Phil Woodland. (2026). Protecting Bystander Privacy via Selective Hearing in Audio LLMs.
Questions for authors
See weakness
3 comments
Q2. Headline result is synthetic and partly an artifact
All switch audio is TTS, with no real bystander recordings and no real-world transfer test. The switch condition changes source, distance, and a 600 ms gap simultaneously, and the paper's own Table 13 shows that with scene-consistent rendering, different/far/600 muting drops from 91.25% to 46.25% and same/far from 60% to 6.25%, meaning roughly half of the 91.3% headline reflects a context-trigger rendering discontinuity rather than distance. The split is item-level only (voices, templates, and tools are shared), and post-SFT self-talk (52%) and side-talk (68%) suggest the learned gate is specific to the switch distribution. The learnability claim is therefore supported only within the benchmark's own synthetic data.
A2. We thank the reviewer for this detailed assessment. We agree with most of it, and much of it is already our own analysis. Table 13 reports exactly the scene-consistent rendering numbers the reviewer cites, and we state in the paper that the distance effect is tied to a rendering discontinuity. We do not claim distance-invariant speaker verification, nor do we claim real-world transfer. Our learnability claim is narrower: the post-trained gate responds to the speaker-source change independently of the other cues, while its distance and boundary sensitivity is scene-dependent.
To keep the headline from being misread, we will revise the text to foreground the artifact analysis and to present the 91.25% figure together with the scene-consistent 46.25% rather than in isolation. We also added full-corpus evaluation on newer models, where qwen3.5-omni-flash mutes 8.0% of switched commands and gpt-audio-1.5 mutes 0.0%, confirming that the failure is not tied to one model. We acknowledge that the split is item-level and that the post-SFT self-talk and side-talk numbers show the gate is specific to the switch distribution, and we will scope the conclusion to the benchmark accordingly.
Q3. Limited evaluation rigor and missing baselines
Results come from one run and one split with 80 held-out switch commands, so the GRPO gain (73→74) is a single item and no significance tests are reported. There is no cascade baseline (speaker verification + ASR + LLM), so the need for end-to-end gating is not shown. The side-talk judge has no reported human agreement, the self-talk family has no positive control (an always-mute policy scores 100%), and the audio corpus is unavailable at review time.
A3. We thank the reviewer for these observations and take them as clear limitations of the present study. We agree that the results come from one run and one split with 80 held-out switch commands, and that the GRPO change from 73 to 74 is a single item. We do not draw a strong claim from this difference and describe the GRPO stage as exploratory in the paper. Adding further runs, significance tests, a second split, a cascade baseline, human agreement for the side-talk judge, a self-talk positive control, and a public audio release would all strengthen the evaluation, and we agree each is worthwhile. These are beyond the scope of the current version, and we have framed our contribution accordingly as a controlled diagnostic of the native action policy rather than a fully validated deployment system. We will state these limitations explicitly in the revision instead of overclaiming, and we will clarify in the discussion that dedicated speaker verification, diarization, and DSP front ends are complementary system-level mechanisms to the native gating studied here.
Q1. Missing comparison with closely related prior work (TPI-Bench Janus-Test, [1], [2])
The central device, holding the transcript fixed while changing only speaker acoustics and training on counterfactual pairs to suppress text-based shortcuts, is the design of TPI-Bench's Janus-Test and speaker-aware hard negatives (e.g., [1][2]). The shortcut finding, the retention check on a general VA benchmark, and the negative result for prompting also appear there. Although the scenario differs (the bystander issues the command, the output is a tool call), the design philosophy and message are the same, yet the paper does not cite this work; Related Work instead names VoxParadox as the closest design. Moreover, [1] covers substantially more: an actionable/ignorable policy, 88K training and 4K diverse test items, a real-world test set, human evaluation, and significance tests. The authors need to clarify the omission of citations and what difference should readers learn from this manuscript.
References cited by the reviewer:
A1. We thank the reviewer for pointing us to this work, which we should have cited. We agree that the high-level design philosophy overlaps with Janus-Test and with [1] and [2]: holding the transcript fixed, changing speaker acoustics, training on counterfactual pairs, and observing that prompting alone does not fix the problem. We will add [1], [2], and TPI-Bench to Related Work and position VGBench against them explicitly rather than naming VoxParadox as the only closest design.
The differences we want readers to take away are the following. First, our output is an executable action in a shared space of Mute, Tool, and Answer, so we measure whether acoustic context changes a decision to act, not a perceptual label or a free-form response. Second, every mute score is paired with same-speaker and text-only tool-selection controls, so an always-mute policy cannot appear strong. Third, the benchmark spans three addressedness families, two of which use real recordings. Fourth, the factorized cue study (Tables 3 and 11) and the cue-balanced SFT control (Tables 12 and 13) decompose the learned gate into source, distance, and boundary effects, which is the diagnostic contribution of this manuscript. We will add a short comparison that makes these distinctions explicit and acknowledge the larger scale and real-world evaluation of [1].