acceptodds
Under review as a conference paper at ICLR 2027

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

Abstract

Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, whose weights cannot be trained and whose per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent user the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since most rollout groups fail identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and lifting an open-weights model above proprietary systems such as GPT-Realtime-2 and Gemini-Live, while using fewer tokens than before training. Along the way it acquires repair behaviours no reward names, such as asking a user to spell a misheard name, and the same checkpoint, which never saw audio question answering, improves on three public audio reasoning and understanding benchmarks.

Primary area:
reinforcement learning
Keywords:
voice agents, audio-native reinforcement learning, tool-using agents, GRPO, reward shaping, omni-modal models, agentic RL, speech

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_86yTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SpeechGym, an audio-native environment for training, not only evaluating, voice agents with on-policy RL. Two Qwen3-Omni-30B-A3B instances converse in native audio: a frozen user model πU\pi_U speaks through its own Talker, and a trainable agent πθ\pi_\theta consumes user audio and textual tool results, reasons in text, and at each step either emits a structured tool call or speaks. The environment is instantiated on τ2\tau^2-bench (Airline, Retail, Telecom) and inherits its tools, databases and outcome verifier unchanged. Only a rank-8 LoRA on the Thinker is trained (GRPO, β=0\beta=0, K=4K=4, Tmax⁡=50T_{\max}=50).

The contribution is primarily environment/infrastructure (a local, API-free, trainable audio-native agentic loop), and secondarily a training recipe (per-turn shaping for sparse multi-turn speech RL).

Critical strengths

Existing voice-agent benchmarks (τ-Voice, VoiceAgentBench, Full-Duplex-Bench-v3) can measure voice agents but not train them. A local, open, zero-API-cost loop that combines native audio, multi-turn tool use and RL, built on an unmodified verifier so that rewards and benchmark scores coincide, is a practical enabler for the community.

Critical weaknesses
  1. The generalisation claims are broader than the evidence supports.

    • Same-lineage, same-domain evaluation. τ-Voice is the voice wrapper of τ2\tau^2-bench (same authors), and evaluation uses the same three domains, tools, policies and success check used in training. What changes is the user simulator and the audio front end (a cascaded TTS user and telephony-grade audio, App. B.3), which is a meaningful shift in the acoustic channel. However, it is held-out-task, held-out-user evaluation within one task distribution, not transfer to "a new, independent benchmark" in the sense the abstract and conclusion suggest. The paper also does not state whether τ-Voice's tasks are disjoint from the τ2\tau^2-bench training split it trains on (Sec. 4.1); this needs to be explicit.
    • Single model, self-play user. Agent and user are the same checkpoint, so the 32% mishearing rate and all training signal come from the agent's own model family's synthesized speech, with no human speech, noise, accents or far-field conditions. The τ-Voice result partly mitigates this, but the paper's central claim for a gym, that it is "designed to accept ... any open omni-modal model" (Contribution 1), is never exercised with a second agent or user model. As a result, reusability, the main selling point of an environment paper, is asserted rather than demonstrated.
    • Framing of the proprietary comparison. Fig. 2 (right) uses leaderboard numbers the authors did not re-run ("not a controlled head-to-head comparison", Sec. 4.4). The trained model (53) is above GPT-Realtime-2 (42), the cascaded baseline (31), GPT-Realtime-1.0 (30) and Gemini-Live-2.5 (26), but clearly below Grok-Voice (67), which appears in the figure and is left out of the prose. Given the uncontrolled setup, the abstract's "lifting an open-weights model above proprietary systems" should be qualified accordingly.
  2. What drives the gain is not fully isolated.

    • No imitation baseline. The ablations compare RL variants (outcome-only, text-mode), but there is no SFT/rejection-sampling fine-tuning baseline on successful audio-native rollouts. Without it, one cannot tell how much of the 24→53 gain requires RL rather than exposure to in-channel successful trajectories.
    • Shaping is not potential-based, and its known failure mode is not measured directly. The authors acknowledge that the shaped return "can favour a long failing rollout with many successful calls over a short successful one" (App. B.2.3; Secs. 3.5 and 4.3). The natural exploit, inflating cheap successful read calls to collect +0.1 each, is checked only indirectly (fewer tokens, fewer dead loops, +5 points on "efficient solve" in Fig. 6).
Questions for authors
  1. Which τ-Voice tasks are evaluated, and are they disjoint from the τ2\tau^2-bench training split used in SpeechGym? If any overlap exists, please report results on the disjoint subset.
  2. Please report the per-episode distribution of tool calls, split into read and write calls, before and after training. Does the policy issue more cheap successful read calls than the base model?
9 comments
AuthorsOpenReview ↗

Reply to Reviewer 86yT [Part 1]

We sincerely thank you for the thorough and constructive review. Your feedback has been invaluable in sharpening the scope of our evaluation and the analysis of our reward. We greatly appreciate your recognition that SpeechGym is "a practical enabler for the community", and that "a local, open, zero-API-cost loop that combines native audio, multi-turn tool use and RL, built on an unmodified verifier so that rewards and benchmark scores coincide" fills the gap left by voice-agent benchmarks that "can measure voice agents but not train them".

We understand your concerns focus primarily on the scope of the generalisation claims and the overlap between τ-Voice and the training tasks, training on one model family's synthesized speech, the framing of the proprietary comparison, the comparison with an imitation baseline, and a direct measurement of the shaping's failure mode.

To address these thoroughly, we have (1) stated the two levels at which τ-Voice is held out from training, a new user simulator, voices and audio channel on all its tasks and, on its test-split tasks, new tasks as well (Sec. 4.1, new App. F.1); (2) scoped the leaderboard statement in the abstract to several proprietary systems on τ-Voice's leaderboard and stated the trained agent's position among the entries shown in Fig. 2, naming Grok-Voice as ahead among them (introduction, Sec. 4.4, conclusion); (3) stated that τ-Voice shows transfer beyond the synthesised training speech, to other voices and a noisy telephony channel (App. A.2); (4) added the individual τ-Voice evaluation runs and MMAU (test-mini) (new App. F.3 and F.4); and (5) reported τ-Voice separately on its 100 test-split tasks, which SpeechGym never trains on, and at each level of held-out evaluation, and added rejection-sampling fine-tuning as an imitation baseline (new Sec. 4.10 and App. G), together with the read and write calls per episode (new App. G.7). All additions have been integrated into our revised manuscript, and their main results are summarized in new Secs. 4.9 and 4.10 of the main text. Below, we address each specific question systematically.

AuthorsOpenReview ↗

Reply to Reviewer 86yT [Part 2]

Q1: The generalisation claims are broader than the evidence supports. Same-lineage, same-domain evaluation. τ-Voice is the voice wrapper of τ2\tau^2-bench (same authors), and evaluation uses the same three domains, tools, policies and success check used in training. What changes is the user simulator and the audio front end (a cascaded TTS user and telephony-grade audio, App. B.3), which is a meaningful shift in the acoustic channel. However, it is held-out-task, held-out-user evaluation within one task distribution, not transfer to "a new, independent benchmark" in the sense the abstract and conclusion suggest. The paper also does not state whether τ-Voice's tasks are disjoint from the τ2\tau^2-bench training split it trains on (Sec. 4.1); this needs to be explicit.

A1: Thank you for this important point. We would like to clarify that on τ-Voice the user simulator, the voices, the audio channel and the harness are held out, while the task distribution is τ²-bench's. The user is τ-Voice's cascaded simulator, in which GPT-4.1 writes each user turn and ElevenLabs v3 speaks it [1], rather than the frozen Qwen3-Omni user the agent trained with, the channel is telephony-grade G.711 μ-law audio at 8 kHz [1] rather than clean model-native speech, and the conversation is run and scored by τ-Voice's own harness (App. B.3). Following your suggestion, the abstract, the introduction, Secs. 4.1 and 4.4, App. A and the conclusion now describe τ-Voice in these terms.

Disjointness. τ-Voice's standard set is the base task set of each domain (τ²-bench's base split), i.e., the union of τ²-bench's training split (178 tasks), on which SpeechGym trains, and its test split (100 tasks), which SpeechGym never trains on. τ-Voice is therefore held out at two levels: on the training-split tasks the user simulator, voices, audio channel and harness are new, and on the test-split tasks the tasks are new as well. Sec. 4.1 now states this, the new App. F.1 gives the counts per domain, and the results on the test-split tasks alone are as follows (Sec. 4.10, App. G.1):

τ-Voice average pass@1 (%)All tasks (278)New user, voices and channel (178)New user and new task (100, disjoint)
Grok-Voice67.366.170.0
Ours (after SpeechGym)52.851.755.0
GPT-Realtime-242.440.047.5
Cascaded baseline31.231.530.8
GPT-Realtime-1.030.427.335.8
Gemini-Live-2.525.826.225.8
Ours before training (base)24.423.525.8

On the 100 disjoint tasks, on the macro average, the trained agent more than doubles the base model, with a gain comparable to that on the other 178 tasks, and keeps its position among the leaderboard entries (App. G.1).

Transfer beyond the agentic task distribution. It is measured separately on audio benchmarks that share no tasks, tools or interactive dialogue with τ²-bench, only audio questions the agent never trained on (Table 4). To further address your concerns, we add a fourth such benchmark, MMAU (test-mini) [2], scored in the same evaluation of the submitted checkpoints that produced Table 4, under the same protocol, and we have added it as App. F.4:

MethodMMAU (test-mini) [2]Δ vs. base
Base Qwen3-Omni-30B-A3B72.60
Outcome-only76.30+3.70
SpeechGym (ours)76.70+4.10
AuthorsOpenReview ↗

Reply to Reviewer 86yT [Part 3]

Q2: Single model, self-play user. Agent and user are the same checkpoint, so the 32% mishearing rate and all training signal come from the agent's own model family's synthesized speech, with no human speech, noise, accents or far-field conditions. The τ-Voice result partly mitigates this, but the paper's central claim for a gym, that it is "designed to accept ... any open omni-modal model" (Contribution 1), is never exercised with a second agent or user model. As a result, reusability, the main selling point of an environment paper, is asserted rather than demonstrated.

A2: Thank you for this insightful comment. We would like to clarify that the τ-Voice gain is measured against speech the agent never heard in training, and that SpeechGym's serving interface is model-agnostic.

Self-play speech. The user model is frozen throughout training (App. A.1), so the agent cannot co-adapt with it into a private protocol; and every τ-Voice number we report is measured against a different speech source: a cascaded simulator (GPT-4.1 writes each user turn, ElevenLabs v3 speaks it) over a telephony channel, in τ-Voice's default Realistic condition, i.e., a different speaker set, a different synthesis system, a degraded 8 kHz codec, and background and burst noise, muffling, frame drops, diverse accents and user interruptions (App. F.2), none of which were seen in training. To further address your concerns, we report the four τ-Voice evaluation runs behind App. D.1 individually, which App. D.1 summarises as avg@4 ± SE, and we have added them as App. F.3:

τ-Voice pass@1 (%), per evaluation runAirlineRetailTelecomOverall
Base, run 126.0035.096.1422.41
Base, run 224.0044.744.3924.37
Base, run 318.0048.252.6322.96
Base, run 434.0048.252.6328.29
Ours, run 162.0078.9527.1956.05
Ours, run 260.0074.5628.0754.21
Ours, run 362.0072.8123.6852.83
Ours, run 466.0070.1816.6750.95
Base, avg@4 ± SE (Table 8)25.50 ± 3.3044.08 ± 3.113.95 ± 0.8424.51 ± 1.33
Ours, avg@4 ± SE (Table 8)62.50 ± 1.2674.12 ± 1.8423.90 ± 2.5953.51 ± 1.08

The weakest trained run is above the strongest base run in every domain, so the agent did not learn to understand only its own model family's voice. On τ-Voice's speech, recovery after a mishearing also rises after training (Table 3), and in Fig. 3 the trained agent recovers from a misheard name by asking the user to spell it. Following your suggestion, we have stated in App. A.2 that τ-Voice shows this transfer beyond the synthesised training speech.

Reusability. SpeechGym talks to each side through one serving request per turn, with audio in and text plus speech out (served by vLLM-Omni, Sec. 3.6 and App. B.4). Beyond that serving interface, the user model, the agent model, the tool executor and the verifier are separate components, and the verifier is τ²-bench's own. Using another omni-modal model as the agent or as the user therefore leaves the tasks, tools, reward and verifier unchanged. At evaluation, the trained agent is also exercised against a second user model: on τ-Voice, whose own harness runs the conversation (App. F.2), its user is GPT-4.1 with ElevenLabs v3 speech rather than Qwen3-Omni, and the gain holds there in every domain (Fig. 2). SpeechGym's episodes also train a rejection-sampling fine-tuning baseline (App. G.3), so the environment is not tied to one training algorithm.

Reviewer_zZ8AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpeechGym, a framework that converts a text agentic environment into a local spoken-interaction training loop. A frozen Qwen3-Omni user simulator and a trainable Qwen3-Omni agent communicate through native audio, while tool calls and results remain structured text. Tasks, databases, and terminal success checks are inherited from tau2-bench. Only a LoRA adapter on the agent's Thinker is trained; the speech-synthesis path is frozen.

Critical strengths
  • The paper focuses on building a local, trainable setup for both the user simulator and the agent makes end-to-end voice experimentation possible without relying on closed voice APIs. The jump from 24% to 53% on tau-Voice is substantial, and the performance hierarchy - full training outperforming outcome-only, text-only, and the base model holds up consistently across all three evaluation domains (Table 5).

  • Across four evaluation seeds, the trained checkpoint reliably beats the base model. While this reflects evaluation stability rather than multiple independent training runs, it shows the reported benchmark gains are not just test-time noise.

  • The method reduces unauthorized tool writes from 23% to 10%, cuts down on repetitive failing tool calls, and handles misheard audio more effectively. Importantly, performance improves across both write-heavy tasks and refusal/read-only tasks, proving the model is not just learning a naive blanket-refusal shortcut.

  • On held-out audio-QA benchmarks, the text-trained baseline stays close to the base model while the voice-trained agent shows noticeable transfer. This control effectively demonstrates that the cross-task gains stem from voice-specific optimization rather than general text instruction tuning.

Critical weaknesses
  • Section 4.1 says training uses a tau2-bench split while tau-Voice tests across those same three domains with a new simulator. But the paper never lists task counts, task IDs, splitting criteria, or an overlap check. Switching simulators changes the environment, but it does not prove the model generalizes to unseen tasks, especially since tau-Voice directly builds on tau2-bench.

  • Basic training specs are missing from the main setup table, including total epochs, and rollout counts.

  • The authors needs to clarify the exact tau-Voice benchmark version, the acoustic environment settings, and how the turn-based Thinker-Talker agent interfaces with tau-Voice's full-duplex protocol. Without these, the headline numbers and leaderboard rankings are hard to interpret or reproduce.

  • Rewarding successful tool execution does not equal genuine task progress. Repetitive reads, redundant queries, and valid but unauthorized writes can still pick up positive intermediate credit. Appendix B.2.3 even notes that a long failing trajectory can end up with a higher score than a clean, short success.

  • Lower token counts and better behavioral numbers help, but they do not prove that this reward signal stays aligned across different tasks or reward weights.

  • The outcome-only run discards 84% of training groups while shaped training drops only 0.4%. Because of this, the comparison cannot separate the benefit of the actual reward formulation from the simple advantage of having vastly more gradient updates. A baseline that fixes the sparse-update problem without process rewards is needed to show that reward shaping is actually necessary.

Questions for authors
  • Please provide exact training and evaluation task manifests, counts by domain, dataset versions, and their task-ID/scenario overlap. Does the 53% result hold on strictly held-out task scenarios, not just a changed user simulator?

  • Which tau-Voice acoustic condition and release were used? How does the agent handle overlapping speech, interruption, and tool-call timing? Are the cited leaderboard systems evaluated under precisely that condition?

  • What are the epochs, rollout counts, retained groups, and optimizer updates for each Table 5 condition? Does shaped training retain its advantage when effective updates are matched or an alternative sparse-reward baseline is used? How sensitive is the result to the +/-0.1 coefficient?

  • How were mishearing, unauthorized writes, and recovery annotated, with what sample counts and agreement checks? Which errors can be distinguished from argument hallucination or reasoning mistakes using the recorded audio?

11 comments
AuthorsOpenReview ↗

Reply to Reviewer zZ8A [Part 1]

We sincerely thank you for the thorough and constructive review. Your feedback has been invaluable in further strengthening the presentation of our evaluation protocol and the analysis of our reward. We greatly appreciate your recognition that our local, trainable setup "makes end-to-end voice experimentation possible without relying on closed voice APIs", that "the performance hierarchy - full training outperforming outcome-only, text-only, and the base model holds up consistently across all three evaluation domains", and that the gains on write-heavy and refusal/read-only tasks show "the model is not just learning a naive blanket-refusal shortcut".

We understand your concerns focus primarily on the task split and its overlap with τ-Voice, the training statistics, the τ-Voice protocol and acoustic setting, whether per-call credit tracks genuine progress, separating the reward formulation from the number of effective updates, and how the behavioural markers were annotated.

To address these thoroughly, we have (1) added τ²-bench's official split, the counts per domain and the overlap with τ-Voice's standard set (Sec. 4.1, new App. F.1); (2) stated the two levels at which τ-Voice is held out from training, a new user simulator, voices and audio channel on all its tasks and, on its test-split tasks, new tasks as well (Sec. 4.1, new App. F.1); (3) added the τ-Voice protocol we ran, including its user side, its codec and how our turn-based agent connects to its harness (new App. F.2); (4) stated that τ-Voice's harness runs timing, overlapping speech and barge-in identically for the base and the trained agent (App. A.2); (5) stated that the four runs of App. D.1 measure the evaluation variance of one trained model (App. D.1, new App. F.3); and (6) reported τ-Voice separately on its 100 test-split tasks and at each level of held-out evaluation, and added outcome-only GRPO with dynamic sampling and a retraining with half and double the per-call credit (new Sec. 4.10 and App. G). All additions have been integrated into our revised manuscript, and their main results are summarized in new Secs. 4.9 and 4.10 of the main text. Below, we address each specific question systematically.

On the four evaluation runs of App. D.1. Following your note in the strengths, we have stated in App. D.1 that its four runs measure the evaluation variance of one trained model, not the variance across independent training runs, and the new App. F.3 lists them individually:

τ-Voice pass@1 (%), per evaluation run (App. F.3)AirlineRetailTelecomOverall
Base, run 126.0035.096.1422.41
Base, run 224.0044.744.3924.37
Base, run 318.0048.252.6322.96
Base, run 434.0048.252.6328.29
Ours, run 162.0078.9527.1956.05
Ours, run 260.0074.5628.0754.21
Ours, run 362.0072.8123.6852.83
Ours, run 466.0070.1816.6750.95
Base, avg@4 ± SE (Table 8)25.50 ± 3.3044.08 ± 3.113.95 ± 0.8424.51 ± 1.33
Ours, avg@4 ± SE (Table 8)62.50 ± 1.2674.12 ± 1.8423.90 ± 2.5953.51 ± 1.08

The weakest trained run is above the strongest base run overall and in every domain.

AuthorsOpenReview ↗

Reply to Reviewer zZ8A [Part 2]

Q1: Section 4.1 says training uses a tau2-bench split while tau-Voice tests across those same three domains with a new simulator. But the paper never lists task counts, task IDs, splitting criteria, or an overlap check. Switching simulators changes the environment, but it does not prove the model generalizes to unseen tasks, especially since tau-Voice directly builds on tau2-bench.

A1: Thank you for this important point. We would like to clarify that the splits are τ²-bench's official ones [1], and that τ-Voice's standard English task set (the set behind its leaderboard) is the base task set of each domain (τ²-bench's base split), i.e., the union of the two splits:

Domainτ²-bench train splitτ²-bench test splitτ-Voice standard set (reported in the paper)
Airline302050
Retail7440114
Telecom7440114
Total178100278

SpeechGym trains on the 178 training-split tasks only, so τ-Voice is held out at two levels: on the training-split tasks the user simulator, voices and audio channel are new, and on the 100 test-split tasks the tasks are new as well. Following your suggestion, we have stated this in Sec. 4.1 and added these counts as App. F.1. We report τ-Voice at each level by splitting the per-task outcomes of the same evaluation runs by τ²-bench's split files, with the leaderboard entries recomputed from their public per-task results in the same way (Sec. 4.10, Fig. 4, App. G.1):

τ-Voice average pass@1 (%)All tasks (278)New user, voices and channel (178)New user and new task (100, disjoint)
Grok-Voice67.366.170.0
Ours (after SpeechGym)52.851.755.0
GPT-Realtime-242.440.047.5
Cascaded baseline31.231.530.8
GPT-Realtime-1.030.427.335.8
Gemini-Live-2.525.826.225.8
Ours before training (base)24.423.525.8

On the 100 tasks never seen in training, on the macro average, the trained agent more than doubles the base model and stays ahead of every leaderboard entry it surpasses on the full set, so the gain does not come from having seen the tasks (App. G.1).

Q2: Basic training specs are missing from the main setup table, including total epochs, and rollout counts.

A2: Thank you for this question. We would like to clarify that all trained conditions of Table 5 start from the same base checkpoint and share the task suite, group size, step budget, optimiser settings (Table 6) and number of training epochs; each differs from the full recipe only in the ablated component. The shared settings are:

Training settingFull recipe (ours)Outcome-onlyText mode
Training tasksτ²-bench training split, 178 tasks (Sec. 4.1, App. F.1)samesame
Training epochs7 (Table 6)77
Rollouts per groupK = 4K = 4K = 4
Groups per epochone per training task (Sec. 3.4, App. A.1)samesame
Step budget per rolloutT_max = 50T_max = 50T_max = 50
Rewardoutcome + per-turn creditoutcome onlyoutcome + per-turn credit
Channelspeechspeechtext
Groups carrying gradient (σ > 0, Table 1)99.6%16%same reward as ours
Optimiser updatesone gradient step per retained group (Algorithm 1, App. C)samesame

Each retained group yields exactly one gradient step (Algorithm 1, App. C), so the updates per collected group equal the retained share of Table 1. Following your suggestion, we have added the number of training epochs to Table 6.

AuthorsOpenReview ↗

Reply to Reviewer zZ8A [Part 3]

Q3: The authors needs to clarify the exact tau-Voice benchmark version, the acoustic environment settings, and how the turn-based Thinker-Talker agent interfaces with tau-Voice's full-duplex protocol. Without these, the headline numbers and leaderboard rankings are hard to interpret or reproduce.

A3: Thank you for this question. We would like to clarify that we ran τ-Voice's standard protocol, with the base and the trained agent connected to its harness in exactly the same way. Following your suggestion, we have added a paragraph on this protocol as App. F.2 (task counts in App. F.1); its content is as follows.

Version and setting. We evaluate on τ-Voice's standard English task set (Airline 50, Retail 114, Telecom 114 tasks) [2] under the leaderboard's standard protocol: its standard user simulator (version v1.0, as listed on the leaderboard), one trial per task and pass@1, with the acoustic setting left at that protocol's default, the Realistic condition (τ-Voice's "regular" speech complexity: background and burst noise, muffling, frame drops, diverse accents, backchannels and user interruptions, over G.711 μ-law 8 kHz telephony). The user side is τ-Voice's own cascaded simulator: GPT-4.1 writes each user turn, ElevenLabs v3 speaks it in the persona's voice and accent, and τ-Voice's channel adds the noise, muffling and frame drops above and passes the audio through a telephony codec, G.711 μ-law at 8 kHz, before it reaches the agent [2].

Interface with the full-duplex harness. τ-Voice's harness runs the conversation, its timing and the scoring, and the base and the trained agent are connected to it in exactly the same way, differing only in the LoRA adapter, so the base-versus-trained comparison is controlled. Our agent is turn-based: at each step it either issues a tool call, which executes without consuming a conversational turn, or speaks (Sec. 3.2); overlapping speech and barge-in are handled by τ-Voice's harness.

Q4: Rewarding successful tool execution does not equal genuine task progress. Repetitive reads, redundant queries, and valid but unauthorized writes can still pick up positive intermediate credit. Appendix B.2.3 even notes that a long failing trajectory can end up with a higher score than a clean, short success.

A4: Thank you for this insightful comment. We would like to clarify that we verify the effect of the per-call credit on τ-Voice, where the score is the unmodified outcome check: repeated identical calls, tokens per task and unauthorised writes all fall after training, while read calls rise by fewer than one per episode as the agent solves more than twice as many tasks.

  • Valid but unauthorised writes earn the per-call credit, yet they fall by more than half after training (Table 3), consistent with the terminal outcome, which such a write fails (Sec. 4.2), outweighing the per-call credit.
  • Repetitive reads and redundant queries: one direct form is repeating the same call. The markers in Fig. 7 (Fig. 6 of the submission) that track repeated calls, counted on every task for both agents, both fall after training, and tokens per task fall as well (App. D.2):
τ-Voice marker (Fig. 7), counted on every task for both agentsΔ after training (points of tasks)
Tool dead loop (an identical call repeated at least 4 times)−5
Severe loop (an identical call repeated at least 8 times)−4

To further address your concerns, we count the read and write calls per episode by τ²-bench's tool types (App. G.7), which shows that reads rise only slightly while the agent solves more than twice as many tasks:

τ-Voice, all 278 tasks (App. G.7)BaseOurs
Read calls per episode4.234.57
Write calls per episode0.710.79
pass@1 (%)24.452.8
  • Long failing versus short successful episodes: this ordering (App. B.2.3) requires a failing rollout that makes far more successful calls than a short success whose calls succeed: with the paper's credit and γ = 0.99, 14 successful calls from its first step on, against a success that makes three successful calls at its first three steps and receives its outcome at its eighth step. This is far above the read and write calls per episode in the table above, and long episodes become rarer after training (Fig. 7, App. D.2).
Reviewer_YzVZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SpeechGym, a local, audio-native environment for reinforcement learning of voice agents on multi-turn dialogue and tool-use tasks. It retains the tasks, tools, and terminal success criteria of τ²-bench while replacing text interaction with native speech. The authors train Qwen3-Omni using GRPO with per-turn tool-execution rewards alongside terminal task rewards. The trained model improves task success on τ-Voice from 24% to 53% and achieves gains on three audio understanding and reasoning benchmarks. The main contributions are the trainable environment, the analysis of speech-related failures, and the experimentally validated training recipe.

Critical strengths

The research problem is meaningful and practically relevant. The paper addresses an important gap between evaluating voice agents and training them. SpeechGym provides a locally executable framework for studying multi-turn native speech interaction and tool use with trainable, open-weight models. Compared with evaluation-only frameworks, an environment that directly supports policy optimization has clear research and practical value. The experimental validation is reasonably comprehensive and closely aligned with the research questions. It includes modality and reward ablations, evaluation with a different user simulator, behavioral analyses, repeated evaluations across seeds, and additional audio benchmarks. These complementary experiments demonstrate improvements in task success and examine the associated behavioral changes, providing substantial support for the usefulness of the environment and training recipe.

Critical weaknesses

The algorithmic novelty is limited. The method primarily combines existing GRPO and reward-shaping techniques, adapting established reinforcement learning practices from other domains to speech agents rather than introducing a new training algorithm. Its contributions lie more in constructing an audio-native training environment, integrating existing components, and conducting empirical analyses. Thus, despite the meaningful problem and reasonably comprehensive experiments, the methodological novelty remains limited.

Questions for authors

Complex tasks may admit multiple valid solution paths with different numbers of tool calls and exploration strategies. Does per-call reward shaping introduce a preference for longer paths containing more successful calls, or penalize necessary exploration involving failed calls? Appendix B.2.3 acknowledges that a long unsuccessful trajectory can receive a higher shaped return than a short successful one. Have the authors quantified how often this occurs and its effect on final task success, particularly for more complex tasks? Have the authors examined robustness across different levels of acoustic degradation and capability retention during extended reinforcement learning training? Specifically, could adaptation to the simulated speech distribution affect general text reasoning or audio understanding under adverse acoustic conditions? The reported audio benchmark improvements are encouraging, but they do not directly establish retention of general text capabilities or robustness to progressively more severe acoustic degradation.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer YzVZ [Part 1]

We sincerely thank you for the thorough and constructive review. Your feedback has been invaluable in clarifying the contribution of our work and the behaviour of its reward. We greatly appreciate your recognition that "the research problem is meaningful and practically relevant", that our work "addresses an important gap between evaluating voice agents and training them", and that "the experimental validation is reasonably comprehensive and closely aligned with the research questions".

We understand your concerns focus primarily on the algorithmic novelty, whether per-call shaping favours longer paths or penalises necessary exploration, and robustness to acoustic degradation together with capability retention.

To address these thoroughly, we have (1) clarified that SpeechGym contributes a trainable environment, a measured diagnosis and a training regime that together open audio-native agentic RL to open models (Sec. 3.5, Tables 1, 2 and 5); (2) shown from the τ-Voice markers and the per-domain results that long and looping episodes become rarer after training and that the per-call credit does not lead the agent away from necessary exploration (Fig. 7, Tables 3 and 5, new App. G.7), and retrained with the per-call credit halved and doubled (new App. G.5); and (3) added MMAU (test-mini) to the audio benchmarks evaluated away from the training audio, which we have integrated into our revised manuscript as the new App. F.4. The main results of these additions are summarized in new Secs. 4.9 and 4.10 of the main text. Below, we address each specific question systematically.

Q1: The algorithmic novelty is limited. The method primarily combines existing GRPO and reward-shaping techniques, adapting established reinforcement learning practices from other domains to speech agents rather than introducing a new training algorithm. Its contributions lie more in constructing an audio-native training environment, integrating existing components, and conducting empirical analyses. Thus, despite the meaningful problem and reasonably comprehensive experiments, the methodological novelty remains limited.

A1: Thank you for this comment. We would like to clarify that SpeechGym's contribution is a trainable environment, a measured diagnosis and a training regime that together open audio-native agentic RL to open models: the per-turn shaping follows classical reward-shaping practice (Sec. 3.5), and we use the existing GRPO optimiser and τ²-bench's own verifier, so that the gains can be attributed to the environment and its densified reward. The contributions are:

  1. A training environment that did not exist. Existing voice-agent benchmarks evaluate an agent but do not train it; τ-Voice, the closest to our setting, benchmarks proprietary real-time agents and runs its user through paid services (GPT-4.1 writes each turn, ElevenLabs v3 speaks it), so every rollout is billed and the price precludes on-policy RL (Sec. 2, App. A.1). SpeechGym is, to our knowledge, the first environment that combines audio-native interaction, multi-turn tool use and RL training of the agent (Sec. 1): the user, the agent, the tools and the verifier all run locally, so on-policy RL runs at no API cost.
  2. A measured diagnosis of what speech breaks. On the same tasks, tools and reward, speech multiplies slot-value errors and dead loops relative to text (Table 2), and these failures cascade into failed calls; this tells the community which failures a voice agent must be trained out of.
  3. The finding that audio agentic RL is gradient-starved under the standard reward. Most outcome-only groups carry no gradient, and the per-turn credit makes nearly every group informative (Table 1). This measurement shows that audio agentic tasks sit in exactly the regime where the sparse variant starves, and that the per-turn credit restores the gradient.
  4. An enabling system component. Both models are served with vLLM-Omni, and the trained LoRA is served per request by adapter name, so an updated policy reaches the rollout workers without reloading weights (Sec. 3.6); a training epoch runs several times faster than with Hugging Face transformers on a single pod (Table 7), which turns multi-epoch audio RL from days into hours.
  5. Evidence that training in audio is what matters. The same recipe trained over text lags far behind on τ-Voice and leaves audio understanding close to the base model, while training in speech improves both (Tables 5 and 9).

Environments that open a new training regime, as the Arcade Learning Environment did for RL [1] (Sec. 2), are a recognised contribution type in their own right, and SpeechGym's value is that it opens this regime to open models. Within this regime, on the same base model, the full recipe is ahead of outcome-only GRPO, training over text and the base model in every domain (Table 5).

AuthorsOpenReview ↗

Reply to Reviewer YzVZ [Part 2]

Q2: Complex tasks may admit multiple valid solution paths with different numbers of tool calls and exploration strategies. Does per-call reward shaping introduce a preference for longer paths containing more successful calls, or penalize necessary exploration involving failed calls? Appendix B.2.3 acknowledges that a long unsuccessful trajectory can receive a higher shaped return than a short successful one. Have the authors quantified how often this occurs and its effect on final task success, particularly for more complex tasks?

A2: Thank you for this insightful question. We would like to clarify that on τ-Voice long episodes and repeated calls become rarer after training, read and write calls rise by fewer than one per episode in total while the agent solves more than twice as many tasks, and recovery after a mishearing rises, so the evaluation does not indicate a preference for longer paths or a penalty on necessary exploration; the case of App. B.2.3 requires a failing rollout with far more successful calls than a short success whose calls succeed.

Longer paths. If the shaping had made the agent prefer longer paths with more successful calls, long episodes and repeated calls would become more frequent and tokens per task would rise. The markers counted on every τ-Voice task for both agents, independently of whether the task passes, move the other way: episodes that ramble past 25 turns, run past 35 turns or hit the step budget all become rarer after training, tool dead loops and severe loops both become rarer, and tokens per task fall (Fig. 7, App. D.2). Counting tool calls directly, read and write calls per episode rise by fewer than one in total while the trained agent solves more than twice as many tasks (new App. G.7).

τ-Voice, all 278 tasks (App. G.7)BaseOurs
Read calls per episode4.234.57
Write calls per episode0.710.79
pass@1 (%)24.452.8

Necessary exploration. A single failed exploratory call costs a small credit, an order of magnitude below the terminal reward, so it cannot make a failing rollout outrank a short success whose other calls succeed unless the failing one makes far more successful calls. For reference, against a success that makes three successful calls at its first three steps and receives its outcome at its eighth step, a failing rollout needs 14 successful calls, made from its first step on (derived with γ = 0.99 and the per-call credit of the paper from the discounted return of Sec. 3.5), well above the read and write calls per episode of either agent in the table above (App. G.7), and efficient solves become more frequent after training (Fig. 7). If exploration were penalised in effect, the agent would avoid calls that can fail; instead recovery after a mishearing, which requires such retries, rises after training (Table 3), so the drop in failed calls (Fig. 7) does not come from abandoning retries.

Effect on final task success. To further address your concerns, we retrain with the per-call credit halved and doubled, keeping everything else unchanged. Final task success stays well above outcome-only training on average at both weights, on τ²-bench's held-out test-split tasks and on the full set (new App. G.5).

Per-call credit (App. G.5)Held-out AirlineHeld-out RetailHeld-out TelecomHeld-out Avg.All 278 Avg.
0 (outcome-only)45.062.517.541.740.0
±0.0555.075.025.051.750.2
±0.1 (paper)60.077.527.555.052.8
±0.250.072.522.548.349.2

More complex tasks. Comparing the per-turn credit with outcome-only training per domain, on dual-control Telecom, the hardest domain, it adds a margin similar to that in the other domains on all 278 tasks (Table 5, Sec. 4.8):

τ-Voice pass@1 (%), all 278 tasks (Table 5, App. G.2)AirlineRetailTelecom
Outcome-only48.060.511.4
Full recipe (ours)62.072.823.7
Gain from the per-turn credit+14.0+12.3+12.3
AuthorsOpenReview ↗

Reply to Reviewer YzVZ [Part 3]

Q3: Have the authors examined robustness across different levels of acoustic degradation and capability retention during extended reinforcement learning training? Specifically, could adaptation to the simulated speech distribution affect general text reasoning or audio understanding under adverse acoustic conditions? The reported audio benchmark improvements are encouraging, but they do not directly establish retention of general text capabilities or robustness to progressively more severe acoustic degradation.

A3: Thank you for these suggestions. We would like to clarify that the evaluation measures the trained agent away from its training audio, through τ-Voice's telephony channel and on three audio benchmarks never trained on, and we add a fourth, MMAU, here.

Acoustic degradation. The training audio is clean model-native speech, whereas τ-Voice delivers the user's speech through a telephony codec (G.711 μ-law, 8 kHz) from different voices and a different synthesis system, in its default Realistic condition with background noise, frame drops and interruptions [2], and the gain holds there in every domain and in every evaluation run (Fig. 2, App. D.1, new App. F.3):

τ-Voice pass@1 (%), per evaluation run (App. F.3)AirlineRetailTelecomOverall
Base, run 126.0035.096.1422.41
Base, run 224.0044.744.3924.37
Base, run 318.0048.252.6322.96
Base, run 434.0048.252.6328.29
Ours, run 162.0078.9527.1956.05
Ours, run 260.0074.5628.0754.21
Ours, run 362.0072.8123.6852.83
Ours, run 466.0070.1816.6750.95
Base, avg@4 ± SE (Table 8)25.50 ± 3.3044.08 ± 3.113.95 ± 0.8424.51 ± 1.33
Ours, avg@4 ± SE (Table 8)62.50 ± 1.2674.12 ± 1.8423.90 ± 2.5953.51 ± 1.08

The weakest trained run is above the strongest base run overall and in every domain. Following your suggestion, App. A.2 now names evaluation under progressively heavier channel degradation and on text-only benchmarks as a direct extension.

Audio understanding. To further address your concerns, we add MMAU (test-mini) [3], scored in the same evaluation of the submitted checkpoints that produced Table 4, under the same protocol, and we have added it as App. F.4:

MethodMMAU (test-mini) [3]Δ vs. base
Base Qwen3-Omni-30B-A3B72.60
Outcome-only76.30+3.70
SpeechGym (ours)76.70+4.10

The gains are not confined to the simulated speech heard in training: on MMAR the largest sub-task gain is on non-speech sound, and on MMAU-Pro, among sub-tasks of substantial size, it is largest on spatial audio and sound (Sec. 4.7, App. D.3).

General text capabilities. Every evaluation of the trained agent in the paper uses the checkpoint at the end of training (App. B.1), i.e., after the full RL run. Only a rank-8 LoRA on the Thinker is trained, the speech-synthesis path is untouched, and the learning rate is small and constant (Table 6). At every step the Thinker reasons in text and emits its tool calls as structured text (Sec. 3.2), and after the full RL run the decisions it reasons out in text improve: policy-aware refusals rise (Fig. 7) and unauthorised writes fall by more than half (Table 3); on audio benchmarks that combine listening with reasoning over a question and its options, the agent likewise improves rather than declines (Table 4).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response [Part 1]

Dear Area Chairs and Reviewers,

Thank you for your time and effort in reviewing our work. Below is our global response:

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are particularly encouraged by the recognition that SpeechGym is "a practical enabler for the community" that fills the gap left by benchmarks that "can measure voice agents but not train them" (86yT), that our local, trainable setup "makes end-to-end voice experimentation possible without relying on closed voice APIs" and that "the performance hierarchy - full training outperforming outcome-only, text-only, and the base model holds up consistently across all three evaluation domains" (zZ8A), and that "the research problem is meaningful and practically relevant" and "the experimental validation is reasonably comprehensive and closely aligned with the research questions" (YzVZ).

After careful consideration of the reviewers' comments, we identified the main concerns focusing on: (1) the overlap between τ-Voice and the training tasks, and whether the gain holds on tasks never seen in training (86yT, zZ8A); (2) what drives the gain, i.e., an imitation baseline and a sparse-reward baseline that recovers the lost updates (86yT, zZ8A); (3) whether the per-call credit stays aligned with task success, including its weight, read inflation and longer paths (86yT, zZ8A, YzVZ); and (4) the evaluation protocol, the training statistics and how the behaviour labels are computed (zZ8A).

We have comprehensively addressed these concerns through clarifications and extensive additional experiments in our revision. The new results are reported in new Appendices F and G, and the main ones are summarized in new Secs. 4.9 and 4.10 (Figure 4). Below, we summarize the clarifications and additional experiments contributed during the rebuttal.

Held-out Tasks (Sec. 4.1, App. F.1, Sec. 4.10, App. G.1, G.6): To address concerns about the overlap with the training tasks, we clarified that SpeechGym trains only on τ²-bench's official training split (178 tasks), and that τ-Voice's standard set is the union of that split and the official test split (100 tasks) [1, 2]. τ-Voice is therefore held out at two levels: on every task the user simulator, the voices and the audio channel are new, and on the test-split tasks the tasks are new as well.

To address the concern about generalisation to unseen tasks, we report τ-Voice separately at each level, from the same evaluation runs, with the leaderboard entries recomputed from their public per-task results in the same way:

τ-Voice average pass@1 (%)All tasks (278)New user, voices and channel (178)New user and new task (100, disjoint)
Grok-Voice67.366.170.0
Ours (after SpeechGym)52.851.755.0
GPT-Realtime-242.440.047.5
Cascaded baseline31.231.530.8
GPT-Realtime-1.030.427.335.8
Gemini-Live-2.525.826.225.8
Ours before training (base)24.423.525.8

On the tasks never seen in training, the trained agent, on the macro average, more than doubles the base model and stays ahead of every leaderboard entry it surpasses on the full set, so the gain does not come from having seen the tasks.

We also repeat the four-run analysis of App. D.1 on the tasks never seen in training (App. G.6):

τ-Voice avg@4 ± SE (%)All tasks (278)New user and new task (100)
Base (no RL)24.5 ± 1.327.3 ± 1.1
Ours (after SpeechGym)53.5 ± 1.155.2 ± 1.0
AuthorsOpenReview ↗

Global Response [Part 2]

Stronger Baselines (App. G.3, G.4): To address concerns about what drives the gain, we clarified that outcome-only GRPO already learns from the policy's own successful in-channel rollouts, and that the per-turn credit also provides a learning signal in groups where every rollout fails.

To address the concerns about an imitation baseline and a sparse-reward baseline that recovers the lost updates, we train rejection-sampling fine-tuning (RFT) [3] on the base agent's successful SpeechGym episodes and outcome-only GRPO with dynamic sampling [4], both from the same base checkpoint on the same training tasks for the same number of epochs:

τ-Voice pass@1 (%)Held-out AirlineHeld-out RetailHeld-out TelecomHeld-out Avg.All 278 Avg.
Base (no RL)25.047.55.025.824.4
RFT (new) [3]40.055.010.035.033.3
Outcome-only45.062.517.541.740.0
Outcome-only + dynamic sampling (new) [4]50.062.520.044.243.1
Full recipe (ours)60.077.527.555.052.8

Both new baselines improve on the base model but stay below the full recipe in every domain and well below it on average, so the gain comes from RL with the per-turn credit, not only from exposure to successful trajectories or from more effective updates.

Alignment of the Per-Call Credit (App. B.2.3, D.2, G.5, G.7): To address concerns about the per-call credit, we clarified that on τ-Voice, where the score is the unmodified outcome check, repeated identical calls, tokens per task and unauthorised writes all decrease after training (Fig. 7, App. D.2, Table 3).

To address the concern about the credit's weight, we retrain the full recipe with the per-call credit halved and doubled, keeping everything else unchanged:

Per-call creditHeld-out AirlineHeld-out RetailHeld-out TelecomHeld-out Avg.All 278 Avg.
0 (outcome-only)45.062.517.541.740.0
±0.05 (new)55.075.025.051.750.2
±0.1 (paper)60.077.527.555.052.8
±0.2 (new)50.072.522.548.349.2

To address the concern about read inflation, we count the tool calls of every τ-Voice episode, split by the read and write tool types that τ²-bench assigns to each tool:

τ-Voice, all 278 tasksBaseOurs
Read calls per episode (new)4.234.57
Write calls per episode (new)0.710.79
pass@1 (%)24.452.8

Halving or doubling the credit keeps the agent well above outcome-only training on average, and read calls rise by fewer than one per episode while the agent solves more than twice as many tasks.

Protocol, Training Statistics and Labels (App. F.2, Table 6, App. D.2): To address concerns about reproducibility, we clarified that we follow τ-Voice's standard protocol in its default Realistic condition: GPT-4.1 writes each user turn, ElevenLabs v3 speaks it, and τ-Voice's channel adds noise, muffling and frame drops over G.711 μ-law 8 kHz telephony before the audio reaches the agent; τ-Voice's harness runs the conversation and the scoring identically for the base and the trained agent. All GRPO conditions share the training tasks, the group size, the step budget and the number of epochs (Table 6). Every behaviour label, including those of Tables 2 and 3, is computed by deterministic rules over the logged trajectories, with no human or LLM judgement.

These clarifications and extensive additional experiments, together with the existing results on τ-Voice in every domain and across four evaluation runs, on three audio reasoning and understanding benchmarks never trained on (now four, with MMAU (test-mini) in App. F.4), and the component ablations, sufficiently demonstrate the effectiveness of SpeechGym. We are grateful for the reviewers' detailed feedback that helped us substantially enhance the paper's rigor.

References

[1] Barrès, V., et al. "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." arXiv:2506.07982, 2025.

[2] Ray, S., et al. "τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains." arXiv:2603.13686, 2026.

[3] Yuan, Z., et al. "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models." arXiv:2308.01825, 2023.

[4] Yu, Q., et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." Advances in Neural Information Processing Systems, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.