- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents SpeechGym, an audio-native environment for training, not only evaluating, voice agents with on-policy RL. Two Qwen3-Omni-30B-A3B instances converse in native audio: a frozen user model speaks through its own Talker, and a trainable agent consumes user audio and textual tool results, reasons in text, and at each step either emits a structured tool call or speaks. The environment is instantiated on -bench (Airline, Retail, Telecom) and inherits its tools, databases and outcome verifier unchanged. Only a rank-8 LoRA on the Thinker is trained (GRPO, , , ).
The contribution is primarily environment/infrastructure (a local, API-free, trainable audio-native agentic loop), and secondarily a training recipe (per-turn shaping for sparse multi-turn speech RL).
Critical strengths
Existing voice-agent benchmarks (τ-Voice, VoiceAgentBench, Full-Duplex-Bench-v3) can measure voice agents but not train them. A local, open, zero-API-cost loop that combines native audio, multi-turn tool use and RL, built on an unmodified verifier so that rewards and benchmark scores coincide, is a practical enabler for the community.
Critical weaknesses
-
The generalisation claims are broader than the evidence supports.
- Same-lineage, same-domain evaluation. τ-Voice is the voice wrapper of -bench (same authors), and evaluation uses the same three domains, tools, policies and success check used in training. What changes is the user simulator and the audio front end (a cascaded TTS user and telephony-grade audio, App. B.3), which is a meaningful shift in the acoustic channel. However, it is held-out-task, held-out-user evaluation within one task distribution, not transfer to "a new, independent benchmark" in the sense the abstract and conclusion suggest. The paper also does not state whether τ-Voice's tasks are disjoint from the -bench training split it trains on (Sec. 4.1); this needs to be explicit.
- Single model, self-play user. Agent and user are the same checkpoint, so the 32% mishearing rate and all training signal come from the agent's own model family's synthesized speech, with no human speech, noise, accents or far-field conditions. The τ-Voice result partly mitigates this, but the paper's central claim for a gym, that it is "designed to accept ... any open omni-modal model" (Contribution 1), is never exercised with a second agent or user model. As a result, reusability, the main selling point of an environment paper, is asserted rather than demonstrated.
- Framing of the proprietary comparison. Fig. 2 (right) uses leaderboard numbers the authors did not re-run ("not a controlled head-to-head comparison", Sec. 4.4). The trained model (53) is above GPT-Realtime-2 (42), the cascaded baseline (31), GPT-Realtime-1.0 (30) and Gemini-Live-2.5 (26), but clearly below Grok-Voice (67), which appears in the figure and is left out of the prose. Given the uncontrolled setup, the abstract's "lifting an open-weights model above proprietary systems" should be qualified accordingly.
-
What drives the gain is not fully isolated.
- No imitation baseline. The ablations compare RL variants (outcome-only, text-mode), but there is no SFT/rejection-sampling fine-tuning baseline on successful audio-native rollouts. Without it, one cannot tell how much of the 24→53 gain requires RL rather than exposure to in-channel successful trajectories.
- Shaping is not potential-based, and its known failure mode is not measured directly. The authors acknowledge that the shaped return "can favour a long failing rollout with many successful calls over a short successful one" (App. B.2.3; Secs. 3.5 and 4.3). The natural exploit, inflating cheap successful read calls to collect +0.1 each, is checked only indirectly (fewer tokens, fewer dead loops, +5 points on "efficient solve" in Fig. 6).
Questions for authors
- Which τ-Voice tasks are evaluated, and are they disjoint from the -bench training split used in SpeechGym? If any overlap exists, please report results on the disjoint subset.
- Please report the per-episode distribution of tool calls, split into read and write calls, before and after training. Does the policy issue more cheap successful read calls than the base model?
9 comments
Reply to Reviewer 86yT [Part 2]
Q1: The generalisation claims are broader than the evidence supports. Same-lineage, same-domain evaluation. τ-Voice is the voice wrapper of -bench (same authors), and evaluation uses the same three domains, tools, policies and success check used in training. What changes is the user simulator and the audio front end (a cascaded TTS user and telephony-grade audio, App. B.3), which is a meaningful shift in the acoustic channel. However, it is held-out-task, held-out-user evaluation within one task distribution, not transfer to "a new, independent benchmark" in the sense the abstract and conclusion suggest. The paper also does not state whether τ-Voice's tasks are disjoint from the -bench training split it trains on (Sec. 4.1); this needs to be explicit.
A1: Thank you for this important point. We would like to clarify that on τ-Voice the user simulator, the voices, the audio channel and the harness are held out, while the task distribution is τ²-bench's. The user is τ-Voice's cascaded simulator, in which GPT-4.1 writes each user turn and ElevenLabs v3 speaks it [1], rather than the frozen Qwen3-Omni user the agent trained with, the channel is telephony-grade G.711 μ-law audio at 8 kHz [1] rather than clean model-native speech, and the conversation is run and scored by τ-Voice's own harness (App. B.3). Following your suggestion, the abstract, the introduction, Secs. 4.1 and 4.4, App. A and the conclusion now describe τ-Voice in these terms.
Disjointness. τ-Voice's standard set is the base task set of each domain (τ²-bench's base split), i.e., the union of τ²-bench's training split (178 tasks), on which SpeechGym trains, and its test split (100 tasks), which SpeechGym never trains on. τ-Voice is therefore held out at two levels: on the training-split tasks the user simulator, voices, audio channel and harness are new, and on the test-split tasks the tasks are new as well. Sec. 4.1 now states this, the new App. F.1 gives the counts per domain, and the results on the test-split tasks alone are as follows (Sec. 4.10, App. G.1):
| τ-Voice average pass@1 (%) | All tasks (278) | New user, voices and channel (178) | New user and new task (100, disjoint) |
|---|---|---|---|
| Grok-Voice | 67.3 | 66.1 | 70.0 |
| Ours (after SpeechGym) | 52.8 | 51.7 | 55.0 |
| GPT-Realtime-2 | 42.4 | 40.0 | 47.5 |
| Cascaded baseline | 31.2 | 31.5 | 30.8 |
| GPT-Realtime-1.0 | 30.4 | 27.3 | 35.8 |
| Gemini-Live-2.5 | 25.8 | 26.2 | 25.8 |
| Ours before training (base) | 24.4 | 23.5 | 25.8 |
On the 100 disjoint tasks, on the macro average, the trained agent more than doubles the base model, with a gain comparable to that on the other 178 tasks, and keeps its position among the leaderboard entries (App. G.1).
Transfer beyond the agentic task distribution. It is measured separately on audio benchmarks that share no tasks, tools or interactive dialogue with τ²-bench, only audio questions the agent never trained on (Table 4). To further address your concerns, we add a fourth such benchmark, MMAU (test-mini) [2], scored in the same evaluation of the submitted checkpoints that produced Table 4, under the same protocol, and we have added it as App. F.4:
| Method | MMAU (test-mini) [2] | Δ vs. base |
|---|---|---|
| Base Qwen3-Omni-30B-A3B | 72.60 | |
| Outcome-only | 76.30 | +3.70 |
| SpeechGym (ours) | 76.70 | +4.10 |
Reply to Reviewer 86yT [Part 3]
Q2: Single model, self-play user. Agent and user are the same checkpoint, so the 32% mishearing rate and all training signal come from the agent's own model family's synthesized speech, with no human speech, noise, accents or far-field conditions. The τ-Voice result partly mitigates this, but the paper's central claim for a gym, that it is "designed to accept ... any open omni-modal model" (Contribution 1), is never exercised with a second agent or user model. As a result, reusability, the main selling point of an environment paper, is asserted rather than demonstrated.
A2: Thank you for this insightful comment. We would like to clarify that the τ-Voice gain is measured against speech the agent never heard in training, and that SpeechGym's serving interface is model-agnostic.
Self-play speech. The user model is frozen throughout training (App. A.1), so the agent cannot co-adapt with it into a private protocol; and every τ-Voice number we report is measured against a different speech source: a cascaded simulator (GPT-4.1 writes each user turn, ElevenLabs v3 speaks it) over a telephony channel, in τ-Voice's default Realistic condition, i.e., a different speaker set, a different synthesis system, a degraded 8 kHz codec, and background and burst noise, muffling, frame drops, diverse accents and user interruptions (App. F.2), none of which were seen in training. To further address your concerns, we report the four τ-Voice evaluation runs behind App. D.1 individually, which App. D.1 summarises as avg@4 ± SE, and we have added them as App. F.3:
| τ-Voice pass@1 (%), per evaluation run | Airline | Retail | Telecom | Overall |
|---|---|---|---|---|
| Base, run 1 | 26.00 | 35.09 | 6.14 | 22.41 |
| Base, run 2 | 24.00 | 44.74 | 4.39 | 24.37 |
| Base, run 3 | 18.00 | 48.25 | 2.63 | 22.96 |
| Base, run 4 | 34.00 | 48.25 | 2.63 | 28.29 |
| Ours, run 1 | 62.00 | 78.95 | 27.19 | 56.05 |
| Ours, run 2 | 60.00 | 74.56 | 28.07 | 54.21 |
| Ours, run 3 | 62.00 | 72.81 | 23.68 | 52.83 |
| Ours, run 4 | 66.00 | 70.18 | 16.67 | 50.95 |
| Base, avg@4 ± SE (Table 8) | 25.50 ± 3.30 | 44.08 ± 3.11 | 3.95 ± 0.84 | 24.51 ± 1.33 |
| Ours, avg@4 ± SE (Table 8) | 62.50 ± 1.26 | 74.12 ± 1.84 | 23.90 ± 2.59 | 53.51 ± 1.08 |
The weakest trained run is above the strongest base run in every domain, so the agent did not learn to understand only its own model family's voice. On τ-Voice's speech, recovery after a mishearing also rises after training (Table 3), and in Fig. 3 the trained agent recovers from a misheard name by asking the user to spell it. Following your suggestion, we have stated in App. A.2 that τ-Voice shows this transfer beyond the synthesised training speech.
Reusability. SpeechGym talks to each side through one serving request per turn, with audio in and text plus speech out (served by vLLM-Omni, Sec. 3.6 and App. B.4). Beyond that serving interface, the user model, the agent model, the tool executor and the verifier are separate components, and the verifier is τ²-bench's own. Using another omni-modal model as the agent or as the user therefore leaves the tasks, tools, reward and verifier unchanged. At evaluation, the trained agent is also exercised against a second user model: on τ-Voice, whose own harness runs the conversation (App. F.2), its user is GPT-4.1 with ElevenLabs v3 speech rather than Qwen3-Omni, and the gain holds there in every domain (Fig. 2). SpeechGym's episodes also train a rejection-sampling fine-tuning baseline (App. G.3), so the environment is not tied to one training algorithm.
Reply to Reviewer 86yT [Part 1]
We sincerely thank you for the thorough and constructive review. Your feedback has been invaluable in sharpening the scope of our evaluation and the analysis of our reward. We greatly appreciate your recognition that SpeechGym is "a practical enabler for the community", and that "a local, open, zero-API-cost loop that combines native audio, multi-turn tool use and RL, built on an unmodified verifier so that rewards and benchmark scores coincide" fills the gap left by voice-agent benchmarks that "can measure voice agents but not train them".
We understand your concerns focus primarily on the scope of the generalisation claims and the overlap between τ-Voice and the training tasks, training on one model family's synthesized speech, the framing of the proprietary comparison, the comparison with an imitation baseline, and a direct measurement of the shaping's failure mode.
To address these thoroughly, we have (1) stated the two levels at which τ-Voice is held out from training, a new user simulator, voices and audio channel on all its tasks and, on its test-split tasks, new tasks as well (Sec. 4.1, new App. F.1); (2) scoped the leaderboard statement in the abstract to several proprietary systems on τ-Voice's leaderboard and stated the trained agent's position among the entries shown in Fig. 2, naming Grok-Voice as ahead among them (introduction, Sec. 4.4, conclusion); (3) stated that τ-Voice shows transfer beyond the synthesised training speech, to other voices and a noisy telephony channel (App. A.2); (4) added the individual τ-Voice evaluation runs and MMAU (test-mini) (new App. F.3 and F.4); and (5) reported τ-Voice separately on its 100 test-split tasks, which SpeechGym never trains on, and at each level of held-out evaluation, and added rejection-sampling fine-tuning as an imitation baseline (new Sec. 4.10 and App. G), together with the read and write calls per episode (new App. G.7). All additions have been integrated into our revised manuscript, and their main results are summarized in new Secs. 4.9 and 4.10 of the main text. Below, we address each specific question systematically.