acceptodds
Under review as a conference paper at ICLR 2027

RIVA: An Open Recipe for Agentic Full-Duplex Speech Models

Abstract

Full-duplex speech language models let users interact with an AI assistant as they would with another person. Existing open models, however, struggle to combine native interaction, in which the model itself decides when to listen, speak, or yield, with the language and agentic capabilities of a strong large language model (LLM). Tool use is also typically handled outside the conversational stream, through separate action channels or backend models. We introduce Riva, an open full-duplex speech model in which a pretrained LLM listens, speaks, and acts within a single stream. Every 480 ms, the model processes incoming speech, decides whether to listen or speak, generates text that a lightweight talker renders as speech, and can emit tool calls in its native format while the conversation continues. Three components make this possible under real speech: a pre-training recipe that reduces the speech-text modality gap; Riva-Dialog, approximately 500K skilled, speech-native conversations spanning 13K hours, complemented by Riva-Entity for named entities and structured identifiers; and reinforcement learning over short decision windows, combining GRPO for interaction-level decisions with on-policy self-distillation for exact tool arguments. Riva is competitive with or outperforms prior open full-duplex models across six benchmarks, including Audio MultiChallenge (18.4 vs. 12.8 APR) and Full-Duplex-Bench v3 (42.0 vs. 35.5 P@1), and achieves an average P@1 of 28.6 on multi-minute, multi-tool customer-service interactions in τ²-Voice.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech, Conversational AI, Tool-calling, Full Duplex

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2jG7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Based on my reading of the introduction, the paper primarily aims to enhance tool-calling capabilities of full-duplex speech language model by leveraging the backbone LLM’s native tool-calling functionality. They also opened the detailed recipe of building FD SLMs.

Critical strengths
  • They opened the detailed recipe of building full-duplex speech language models.
Critical weaknesses
  • I disagree with the following statement in the abstract: “Existing open models, however, struggle to combine native interaction, in which the model itself decides when to listen, speak, or yield, with the language and agentic capabilities of a strong large language model (LLM).” There are already many full-duplex spoken dialogue models, including Moshi, PersonaPlex, and Raon-speech, that decide when to speak and when to yield.
  • Raon-speech is also an open-source full-duplex speech language model that adopts a thinker–talker architecture with a lightweight talker. The novelty relative to this model is unclear.
  • Using the backbone’s native tool-calling functionality appears to be the paper’s main novelty, but this seems somewhat weak.
  • The explanation of tool-calling data construction appears insufficient, given that enhancing tool-calling capabilities seems to be the paper’s main contribution.
  • In particular, the description of Riva-Dialog in lines 275–293 does not sufficiently explain the closed set of tools or the format used to call them. These details should be provided in the main text or described thoroughly in the appendix with a corresponding reference.
  • The description of GRPO in lines 303–320 refers to “short decision windows around the relevant event,” but does not specify exactly how many windows or turns are used for RL.
  • In agentic RL, it is necessary to consider whether a particular tool was called within a multi-turn scenario. The credit assignment algorithm is also not clearly described.
  • The paper appears to use several techniques, including noise augmentation and on-policy self-distillation, but the presentation feels scattered. Noise augmentation seems more appropriate for the dataset section, and the paper should establish more clearly what its main contributions are.
  • The evaluation is substantially insufficient. For example, Table 1 reports only TT Fluency, Instr. Foll., and Task Metric for Full-Duplex Bench v2, whereas the official FDB v2 reports TT and IF across four categories: Daily, Correction, Entity Tracking, and Safety.
  • This raises the concern that the table may cherry-pick favorable results. If space is limited, the full evaluation results should at least be included in the appendix.
  • Regarding the writing, too many sentences use colons. In lines 70–89, presenting Representation, Data, and Decisions within a single paragraph makes the text difficult to read.
  • The LoRA rank is not specified in the training description in lines 249–261.
  • In line 290, a space is missing in “markers.5).”
Questions for authors
  • What is the novelty of the proposed model relative to Raon-speech, which also adopts a thinker–talker architecture with a lightweight talker?
  • The thinker’s 480 ms interval seems relatively long compared with the 80 ms inference intervals of Moshi and PersonaPlex. Is 480 ms really appropriate? Would this not introduce a consistent delay when a person converses with the model?
  • What LoRA rank was used for the training described in lines 249–261?
  • For Riva-Dialog in lines 275–293, what closed set of tools was defined, and in what format were these tools called? Please provide these details or add a reference to a detailed explanation in the appendix.
  • In lines 303–320, exactly how many windows or turns are used for RL when applying GRPO to “short decision windows around the relevant event”?
  • How does the RL setup account for whether a particular tool is called within a multi-turn scenario, and what credit assignment algorithm is used?
  • Please consult and cite the following relevant work:
  • Could you provide the full FDB v2 evaluation results, including TT and IF for all four categories, at least in the appendix?
  • I would also like to see results on FDB v1 to assess the model’s full-duplex turn-taking and pause-handling capabilities.
  • Could Representation, Data, and Decisions in lines 70–89 be organized into bullet points rather than presented within a single paragraph?
  • Please add a space between “markers.” and “5)” in line 290.

I reviewed the paper without using AI, so there may be points I have missed. I will raise my score if the authors provide sufficient and convincing explanations. Thank you.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2jG7

We thank the reviewer for the detailed assessment. We understand that you did not find the paper especially informative, as reflected in your rating and we acknowledge that our presentation can be improved to convey our contributions better. The main goal of Riva is to have a single full-duplex speech language model (FD-SLM) that unifies interaction, language capabilities and native agentic capabilities. We show that achieving this goal requires contributions across a non-blocking tool-stream design, pre-training and post-training recipe and new training datasets across both. We have addressed your comments, and added experimental results below, and we would greatly appreciate any further candid feedback.

Improved Presentation

Thank you for this suggestion. We have restructured the abstract, and the introduction to convey our contributions more clearly, across separate paragraphs. We have also moved the noise augmentation to the dataset section.

Clarifying The Abstract Claim

We thank you for pointing this out, and we agree that existing models, including Moshi, PersonaPlex, and Raon-Speech, support native turn-taking. However, they exhibit substantially weaker language capabilities and lack native agentic capabilities. Our main claim was that Riva “combines” native interaction, strong language (intelligence) and agentic capabilities. Which to the best of our knowledge, no other open-source model does. We have revised the abstract to clarify this.

Novelty Of Native Tool-Calling

The challenge is not the tool-call format itself, but making tool calls work inside a full-duplex stream. A complete tool call often takes 2–3 s to generate, during which a naively fine-tuned FD-SLM cannot listen or respond, blocking interaction. Our tool stream design splits calls across micro-turns and interleaves them with the user and assistant streams, so the conversation continues while calls are generated, executed, and their results consumed.

Beyond the stream design, to our knowledge no existing datasets target argument errors from spoken entities, or provide full-duplex tool-use conversations with features like user repairs, disfluencies, and interactive spoken fillers.

Novelty Relative to Raon-Speech

We agree that thinker-talker architectures exist, and in fact our paper states that we “adopt the thinker–talker design”. Our main contributions are a novel tool stream that enables non-blocking, streaming tool calls; a pre-training recipe that narrows the gap to the backbone’s intelligence; post-training recipe that enables conversational capabilities and new training datasets across these stages. Together these lead to a highly capable model; Riva substantially outperforms Raon-SpeechChat on most benchmarks while adding native tool-calling capabilities that it lacks.

Justification For Micro-Turn Size

Prior work such as MiniCPM-o 4.5 [1] shows that temporal granularity has a capacity-latency tradeoff, which we also encounter in our preliminary experiments. Therefore, we choose 480ms as our micro-turn size to balance general capability and latency. Importantly, the inference implementation also substantially affects user perceived latency which we further optimize to improve interactivity (Appendix A.4)

AuthorsOpenReview ↗

Response to Reviewer 2jG7

Additional Results On Full-Duplex Bench

As requested, we report the FDB v1 results and per-category FDB v2 breakdown below. Riva demonstrates competitive turn-taking and substantially better pause and interruption handling. Here Syn/CANDOR represent pause handling.

MethodSyn. TOR↓CANDOR TOR↓BC TOR↓BC Freq.↑BC JSD↓TT TOR↑TT Lat.↓Int. TOR↑Int. Score↑Int. Lat.↓
Moshi1.0001.0000.4910.1790.6840.7140.270.9103.320.97
Raon0.9490.8610.5450.1440.7140.8990.240.9903.010.21
MiniCPM-o 4.50.2480.3660.6100.0130.9300.8400.540.9254.370.73
Ours0.0220.0420.0000.0320.8391.0000.601.0004.560.59
DomainTurn-taking ↑Instr. Following ↑Task ↑
Daily4.403.953.60
Correction3.423.484.40
Entity tracking3.984.063.90
Safety4.284.564.90

Additional Details On RL & References

Given the preceding context of a dialogue, we roll out only the model’s next one turn around a specific event, such as an interruption, a tool call, or a turn transition, typically 4–8 s. The rewards are computed within this window, and since each reward applies directly to the single decision that produced it (such as a tool-call or turn-taking), we do not need a multi-turn credit assignment algorithm.

Moreover, we now reference the three papers shared, in Sec 3.2.

Tool Format And Environments

We use the Hermes tool-call format used by Qwen3, and now describe this Appendix A.1 The full set of the environments and the tools we use are now presented in Table 6 in the Appendix C.2.

Low Rank Implementation Details

We used a LoRa rank of 64, and did not find sufficient difference in 32/64/128 in initial preliminary experiments. We have added this information in Training Details.

[1] Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction

Thank you for the detailed response and clarifications. While several of my concerns regarding evaluation and implementation details have been addressed, my main concern about novelty remains. In particular, the architectural novelty over Raon-Speech appears limited, the 480 ms micro-turn seems more like a design choice than a methodological contribution, and a substantial portion of the contribution lies in the pre-training/post-training recipe and dataset construction. Therefore, I will maintain my current score.

AuthorsOpenReview ↗

Response To Reviewer 2jG7

We thank the reviewer for responding and are glad that several concerns regarding evaluation and implementation have been addressed.

We would like to clarify that we never claimed the thinker-talker architecture as a contribution. Our paper states that we adopt it, and the 480 ms micro-turn size is a design choice.

As noted by Reviewers tmHX and vNfR, our key architectural contribution is the non-blocking tool stream, which maintains full-duplex interaction during tool-call and execution.

We agree with the reviewer that a substantial portion of the contribution is additionally in pre-training, post-training recipe and training datasets, which contribute to our goal:

  • Pre-training recipe: which narrows the speech-text modality gap under streaming input, preserving the backbone's language capabilities, together with Riva-Entity dataset, which improves spoken entity recognition, a prerequisite for accurate tool calls.

  • Post-training recipe: supervised fine-tuning on Riva-Dialog dataset, which jointly supervises general assistant skills and full-duplex interaction, followed by RL over short decision windows that refines interactivity and tool calling.

Our component ablations establish the benefits of these recipes and datasets. Together, these unify native interactivity, strong language capabilities, and tool calling in a single FD-SLM, which outperforms or is competitive with prior open FD-SLMs across six benchmarks.

Reviewer_vNfROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes RIVA, aimed at addressing the difficulty of balancing natural interaction and the complex tool-calling capabilities of large language models (LLMs) in existing open-source full-duplex speech language models (FD-SLMs). The model utilizes a Thinker-Talker architecture, integrating speech inputs, text generation, and tool calling within the Thinker's autoregressive stream using 480ms micro-turns, with speech synthesis handled by the Talker. To achieve this, the authors implement cross-modal alignment pre-training, construct the Riva-Dialog and Riva-Entity training sets, and apply GRPO and on-policy self-distillation reinforcement learning targeted at key interaction and tool-calling decision windows. The paper reports that Riva outperforms the evaluated open-source full-duplex models on agentic benchmarks such as τ²-Voice.

Critical strengths
  • Non-blocking Tool Stream: Riva integrates the <tool_stream> into the 480ms micro-turns, allowing the model to continue generating conversational fillers while simultaneously submitting tool call parameters. This design can reduce silent waiting periods during API calls.
  • Efficient Decision-Window Alignment: The authors use focused decision windows around critical segments like interruptions, backchannels, and tool recognition for RL, rather than full-dialogue rollouts. Combining GRPO with on-policy self-distillation provides additional token-level supervision when exact-match rewards are sparse in identifier recognition tasks.
Critical weaknesses
  • Asymmetric Baseline Evaluation: The core highlight of the paper is its performance on the long-horizon agentic task (τ²-Voice), achieving an average P@1 of 28.6. However, Table 1 marks models without a tool-calling interface (e.g., PersonaPlex, MiniCPM-o 4.5) with “—”. These comparisons do not establish whether the “single-stream full-duplex architecture” offers advantages over a traditional “cascaded architecture” for tool use. The paper lacks a horizontal comparison against a strong cascaded baseline such as Streaming ASR + an open LLM with strong tool-calling capabilities + low-latency TTS, leaving its advantages over such systems unclear.
  • Unaddressed API Latency in Real-time Constraints: The paper describes asynchronous tool execution, with results inserted into the Thinker's context once available and consumed in a subsequent 480ms micro-turn. In real-world commercial applications (e.g., booking tickets, database queries), API response times are often unpredictable and can take several seconds. However, the paper does not report systematic evaluation of how the model maintains full-duplex interaction during such multi-second waits (e.g., if the user suddenly changes their mind or starts chatting during the wait, how does the model respond or handle cancellation?).
  • Hidden Text Bottleneck: Despite being described as a “native” full-duplex stream, the Thinker's core mechanism still relies on generating discrete text tokens to drive the Talker, alongside Thinker hidden states. For semantic reasoning and tool calling, it remains unclear how effectively the model uses non-linguistic acoustic features from the original audio, such as prosody and tone. In this respect, the architecture's advantages over a highly optimized cascaded system remain unclear.
Questions for authors
  1. To assess the contribution of the single-stream design in agentic tasks, can the authors provide a performance comparison on τ²-Voice or Full-Duplex-Bench v3 against at least one cascaded baseline system during the rebuttal phase?
  2. In actual deployment, if a lookup.customer tool call takes 5 seconds to return a result, what state does the system enter under the current 480ms TDM framework? If the user speaks during this period, can the model capture the input and abandon the currently executing tool call?
  3. Have the authors evaluated how acoustic cues, such as hesitation or urgency in the user's tone, affect decisions about whether to interrupt or invoke a specific tool?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vNfR

We thank the reviewer for their careful reading and constructive suggestions. We address each point below with additional experiments and clarifications.

Q1 Comparison Against Cascade

We thank the reviewer for this suggestion. We built a strong cascade system, which consists of Parakeet for ASR, Qwen3-TTS for speech synthesis, VAP [1] for turn-taking management and Qwen3 8B for LLM.

Interactivity The cascade is 3-4 times slower than Riva, and cannot back-channel.

MethodSyn. TOR↓CANDOR TOR↓BC TOR↓BC Freq.↑BC JSD↓TT TOR↑TT Lat.↓Int. TOR↑Int. Score↑Int. Lat.↓
Riva (ours)0.0220.0420.0000.0320.8391.0000.601.0004.560.59
Cascade0.0000.0600.0000.0001.0000.9501.760.9954.692.60

Tool Calling While the cascade retains the LLMs tool-calling capability, we show that Riva is still competitive in accuracy (within 2 points of it) while providing native low-latency interaction. Moreover, in τ2\tau^2-Voice Riva outperforms the cascade baseline, which we attribute to improved spoken entity recognition from Riva-Entity and higher-quality agentic trajectories in Riva-Dialog.

Table 1: FDB v3 Pass@1 and latency (s).

MethodFDB v3 P@1 ↑FDB v3 Latency ↓
Riva (ours)42.00.56
Cascade (Qwen3-8B)44.02.96

Table 2: τ2\tau^2-Voice Pass@1 per domain.

MethodRetail↑Airline↑Telecom↑
Cascade (Qwen3-8B)6.132.023.8
Riva (ours)15.041.029.8

Q2 API Latency Constraints

We thank the reviewer for highlighting this aspect. A key feature of our tool stream is that it maintains interactivity while tool calls are generated and executed. In τ2\tau^2-Voice, for instance, tool calls often take 5–8 s, which our model handles reliably. When a tool call starts, the model first enters a <SPEAK> state with a short acknowledgement, but it can be interrupted, upon which it switches to <LISTEN>.

Since standard evaluation benchmarks don’t cover such interruptions, we conduct a simple experiment, where we use the tool-calling benchmark FDB v3, and during inference when a tool-call is in progress, we interrupt it with another question from AlpacaEval. Using similar interruption metrics as FDB v1 interruption axis, we observe that Riva has a takeover rate (TOR) of 78% and LLM-judge score of 3.87, where Riva responds to interrupting query first, responds to it and later continues the tool-call response.

Q3 Hidden Text Bottleneck

Thank you for raising this concern. Unlike a cascade, whose LLM only sees transcripts, Riva’s thinker reasons directly over continuous speech, retaining cues such as prosody and tone. Distinguishing pauses from turn ends and backchannels from interruptions relies on such cues [2] and Riva handles these tasks reliably in FDB v1 as well as FDB v3 which contains tool-call queries under natural pauses and hesitations (we kindly refer you to FDBv1 and FDBv3 Tables in response to Q1)

To our knowledge, no existing benchmark evaluates how affective cues such as urgency influence tool-calling decisions; we see this as a promising direction for future work.

[1] Voice Activity Projection: Self-supervised Learning of Turn-taking Events

[2] How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models

Reviewer_vNfROpenReview ↗

Official Comment by Reviewer vNfR

Dear authors,

Thank you for the detailed response and the additional experiments conducted during the rebuttal period. The additional cascaded baseline is helpful. The added interruption experiment addresses my concern about interaction during tool execution, although handling changed or withdrawn requests remains to be clarified. I also agree that the Thinker’s continuous audio input does not support characterizing the architecture as having a “text bottleneck,” and consider controlled evaluation of prosodic cues a useful direction for future work. Regarding Q2, a full-duplex conversational frontend coupled with an asynchronous reasoning/tool-use backend, such as KAME [1] and DuplexOmni [2], can also maintain interaction while API calls are pending and incorporate results when available. What additional benefits does RIVA’s unified decision stream offer for handling tool latency, particularly when the user changes their request or a returned result becomes outdated? Or at least discuss the comparison between the two, as it seems that the front-end and back-end brain architecture is also a full duplex agent direction that the community is very concerned about. Discussing this alternative and the associated trade-offs would help clarify the motivation for the proposed design.

[1] So Kuroki, Yotaro Kubo, Takuya Akiba, and Yujin Tang. “KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI.” ICASSP, pp. 19362–19366, 2026. [2] Muye Huang et al. “DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interaction.” arXiv preprint arXiv:2606.09186, 2026.

AuthorsOpenReview ↗

Response To Reviewer vNfR

We thank the reviewer for confirming that the additional experiments address the interactivity concerns and that Riva's architecture does not have a text bottleneck. We are also glad, the cascade baseline comparison helped.

Frontend-backend systems are a viable approach for duplex tool-calling. However, to the best of our knowledge, neither KAME nor DuplexOmni evaluates tool-calling in their research papers. If such systems demonstrate tool-calling, a key challenge would be maintaining coordination between the two models. User interruptions and repairs have to be sent to the backend, and partial responses integrated back into the frontend, which introduces challenges in both latency and maintaining accurate contexts, since the frontend and backend may operate on different partial views of the evolving conversation. In contrast, Riva handles interaction and tool use within a shared context, eliminating this coordination overhead. A unified model like Riva can also be trained end-to-end, which allows interaction and tool use to improve jointly as the model is scaled. We will add a detailed discussion in the revised version of the paper.

Reviewer_vNfROpenReview ↗

Official Comment by Reviewer vNfR

Dear authors,

Thank you for the further clarification. Engineering realization is valuable, but my concerns about the motivation for the proposed design still remains. I iwill increase my soundness score and maintain the rating score.

Reviewer_tmHXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Riva, a full-duplex speech model built on Qwen3-8B that integrates listening, turn-taking, response generation, and tool calls within one autoregressive stream. Its training recipe combines speech–text pretraining, synthetic conversational and entity-focused data, and reinforcement learning over short decision windows. Results show competitive performance across six benchmarks, including long-horizon agentic voice tasks.

Critical strengths
  1. The paper addresses a practical problem: combining natural spoken interaction with strong language capabilities and tool use. Integrating tool calls into the backbone’s native stream is well motivated.
  2. Riva-Dialog and Riva-Entity provide useful, complementary supervision for conversational behavior and accurate spoken tool arguments. The data ablations demonstrate substantial gains.
  3. The paper is clearly presented and evaluates a broad range of capabilities, with strong results against several open full-duplex baselines.
Critical weaknesses

Weaknesses

  1. The contribution of RL is not isolated. Decision-window RL is presented as a main contribution, but there is no comparison between SFT and SFT+RL, or between GRPO alone and GRPO with self-distillation. These ablations are needed to establish which improvements come from RL.

  2. The baseline comparison does not establish the benefit over a strong cascade. A streaming ASR → Qwen3-8B → TTS system, evaluated under the same conditions, would help assess the accuracy, latency, and interaction trade-offs of the proposed design. ChipChat and DuplexCascade are relevant examples of low-latency cascaded interaction. Additional audio tool-use baselines, such as Qwen3-Omni, would also strengthen the evaluation, with differences in duplex capabilities clearly stated.

  3. Validation of the synthetic conversational data is limited. Conversation Analysis grounding and CANDOR-based backchannel matching are useful, but no human audit of interaction naturalness is reported. ASR verification checks spoken content rather than the realism of timing, prosody, or interruptions. A small human audit and quantitative interaction statistics would strengthen confidence in the data.

  4. Speech generation and conversational experience are insufficiently evaluated. The reported metrics emphasize response content, task success, and interaction behavior, without directly assessing generated speech quality. Output intelligibility and acoustic-quality measures, together with a small listening study of naturalness and turn-taking, would provide a more complete evaluation.

  5. Agentic generalization and result reliability remain unclear. Riva receives training in airline, retail, and telecom environments, matching the τ²-Voice evaluation domains. This makes generalization beyond those domains difficult to assess. Held-out-domain evaluation, uncertainty estimates over repeated runs, and an explanation of NVIDIA VoiceChat’s zero scores would strengthen the conclusions.

  6. Understanding beyond words is not evaluated. The evaluation does not establish whether Riva can answer questions about vocal emotion, prosody, or the number of speakers. Assessing these abilities—or explicitly identifying them as outside the scope—would clarify the model’s speech-understanding capabilities.

Questions for authors
  1. How much do decision-window RL and self-distillation each contribute beyond SFT?
  2. How does Riva compare with a same-backbone streaming cascade in task accuracy, latency, and turn-taking?
  3. Was the synthetic interaction data audited by humans, and which judge models were used for benchmark evaluation?
  4. Can you clarify baseline tool integration and provide evidence of agentic generalization beyond the training domains?

Minor Comments

Appendices A.1 and E.1/E.2 contain repeated material. Please standardize τ-Voice/τ²-Voice naming and correct “exlcudes” in Table 3.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tmHX

We sincerely thank the reviewer for the careful reading and constructive suggestions. We address each point below with additional experiments and clarifications.

W1. Ablation Of RL

We use RL to improve two capabilities, interactivity and tool-calling, and add self-distillation specifically to improve argument accuracy in the agentic scenarios where exact match rewards are sparse.

Interactivity (FDB v1). GRPO substantially improves turn-taking decisions. The model better handles pauses, and responds to interruptions more reliably, at the cost of a small latency increase in interruptions.

MethodSyn. TOR↓CANDOR TOR↓BC TOR↓BC Freq.↑BC JSD↓TT TOR↑TT Lat.↓Int. TOR↑Int. Score↑Int. Lat.↓
SFT0.0510.0740.3090.0490.8290.9660.560.933.680.43
SFT + GRPO0.0150.0510.0000.0310.8421.0000.621.004.470.61
SFT + GRPO + OPSD0.0220.0420.0000.0320.8391.0000.601.004.560.59

Tool calling (τ2\tau^2-Voice, P@1). GRPO only yields modest gains, while self-distillation drives most of the improvement in tool-call capability, especially in retail and airline, which involve many spoken entities such as names and IDs.

MethodRetail ↑Airline ↑Telecom ↑
SFT5.035.027.0
SFT + GRPO8.035.025.0
SFT + GRPO + OPSD15.041.029.8

W2. Cascade Baseline Comparison

We built a strong cascade system, which consists of Parakeet for ASR, Qwen3-TTS for speech synthesis, VAP [1] for turn-taking management and Qwen3 8B for LLM.

The cascade is 3-4 times slower than Riva, and cannot back-channel. While the cascade retains the LLMs tool-calling capability, we show that Riva is still competitive in accuracy (within 2 points of it) while providing native low-latency interaction. Qwen3-Omni, being a much larger MoE model, has higher task accuracy, and has 3 times higher latency.

MethodSyn. TOR↓CANDOR TOR↓BC TOR↓BC Freq.↑BC JSD↓TT TOR↑TT Lat.↓Int. TOR↑Int. Score↑Int. Lat.↓
Riva (ours)0.0220.0420.0000.0320.8391.0000.601.0004.560.59
Cascade0.0000.0600.0000.0001.0000.9501.760.9954.692.60
MethodFDB v3 P@1 ↑FDB v3 Latency ↓
Riva (ours)42.00.56
Cascade (Qwen3-8B)44.02.96
Cascade (Qwen3-Omni)48.01.96

W3. Validation Of Synthetic Conversational Data

To validate our synthetic data, we now report both quantitative interaction statistics and a human audit.

Interaction statistics. Following dGSLM [2] we measure per-minute turn-taking events. These are descriptive scores, so we compare them against Fisher as a reference for naturalness and find that they resemble it on most statistics. Our back-channel rates are calibrated against backchannels/min found in the CANDOR dataset, which are lower than Fisher.

DatasetIPU/minPause/minGap/minBackchannels/min
Fisher14.15.14.44.05
Riva-Dialog12.98.53.22.08

Human audit. We adapt the Behavior-SD [3] annotation protocol (2 annotators evaluate 50 conversations from each dataset). As real human conversation, Fisher serves as an upper reference for naturalness and emotional appropriateness, while its lower sound quality reflects its 8 kHz telephone audio. We show that Riva-Dialog has natural dialogues, has emotionally appropriate responses and high-sound quality.

DatasetDialogue Naturalness ↑Emotion Appropriateness ↑Sound Quality ↑
Fisher4.88 ± 0.074.34 ± 0.203.61 ± 0.11
Behavior-SD3.56 ± 0.213.78 ± 0.073.75 ± 0.21
Riva-Dialog4.15 ± 0.253.91 ± 0.184.10 ± 0.16
AuthorsOpenReview ↗

Response to Reviewer tmHX

W4. Speech Generation Quality

We measure speech intelligibility using word error rate (WER) and acoustic quality using automated UTMOS on a random set of 500 samples from the evaluation benchmarks we use. We use human MOS to evaluate the naturalness of turn-taking and spoken output. Riva produces intelligible speech, with high audio quality and naturalness, comparable to the other baselines. (WER/UTMOS on 500 sampled responses, WER: Parakeet ASR against the model’s own output text, MOS: 2 raters and 25 samples per model)

ModelWER ↓UTMOS ↑Human MOS ↑
Raon-SpeechChat9.5%4.144.30
MiniCPM-o 4.54.4%4.294.24
Riva2.7%4.224.38

W5. Generalization To Agentic Benchmarks

Generalization. We additionally evaluate Riva on the multi-turn English split of VoiceAgentBench[4], built on API-Bank, whose domains and APIs are unseen during our training. We compare it with the cascade of Parakeet and Qwen3-8B. The cascade selects tools and follows the schema more reliably, as expected from a text LLM. However, Riva matches it on parameter accuracy (which is the final pass rate) which requires correctly extracting spoken entities, and does so at 3× lower latency.

MethodTool Selection ↑Schema ↑Parameters (Pass Rate) ↑Latency ↓
Parakeet + Qwen3-8B92.686.344.91.57
Riva80.172.346.10.52

Result Clarification. Over three repeated runs, 2-Voice P@1 has a standard deviation of 2, 3, and 6 points for retail, airline, and telecom respectively. The higher-variance in telecom is from its dual control setting, where the user also takes an action leading to longer and more variable conversations. The VoiceChat model scores zero, because 2-Voice has an extremely strict metric, which requires the final database state to match the goal, hence even a single failed tool-call results in a failed task. We observe that the overwhelming majority of errors occur in the initial customer authentication tool-call step, with the rest in subsequent tool calls.

W6. Regarding Paralinguistic Understanding

Thank you for this suggestion. This work focuses on unifying native interactivity, strong language capabilities, and tool use in a single full-duplex speech model. While paralinguistic understanding is out of scope, Riva still processes continuous speech representations that carry cues such as intonation and vocal emotion, therefore training on emotional data is a natural extension of our work.

Q3. Judge Models For Evaluation

We thank the reviewer for raising this. We have now specified the judge model for each benchmark in the revised supplementary material.

Regarding Dataset Provenance

All our voices come from cloning come from Mozilla Common Voice, released under CC0 by contributors who consented to public-domain donation; we comply with its term (Appendix C4). Moreover, the entities themselves are either from public wiki entities or fictional entities generated using LLM. Finally, all our conversations are evaluated through LLM-as-judge to check for helpfulness, truthfulness and safety.

Minor Comments

We have removed the repeated text and standardized τ2\tau^2-Voice

[1] Voice Activity Projection: Self-supervised Learning of Turn-taking Events

[2] Generative Spoken Dialogue Language Modeling

[3] Behavior-SD: Behaviorally Aware Spoken Dialogue Generation with Large Language Models

[4] VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

I thank the authors for the additional experiments, which address several of my concerns and strengthen the paper.

However, my main concerns about agentic performance remain. GRPO alone provides limited tool-calling improvements, and Riva still trails the same-backbone cascade in tool selection and schema accuracy. The relatively low agentic success rates and uncertainty about the robustness of the gains also limit the strength of the claims.

While I appreciate the authors' efforts, I maintain my 2. Weak Reject rating.

AuthorsOpenReview ↗

Response To Reviewer tmHX

We thank the reviewer for responding and are glad the additional experiments addressed several concerns. We would like to clarify your remaining points.

Comparison With Cascade Higher tool-selection and schema accuracy alone do not ensure successful tool use, since even a correctly selected tool can fail when its arguments contain misrecognized spoken entities, a critical issue in voice agents. Riva specifically addresses this issue and outperforms the cascade in final pass rate on VoiceAgentBench. Moreover, the cascade is 4-5x slower, which significantly limits its interactivity.

GRPO Ablation GRPO alone is limited for tool calling because exact-match rewards on spoken identifiers are sparse; this is precisely why our RL recipe adds on-policy self-distillation, which provides denser token-level supervision. Together, they improve τ²-Voice P@1 from 22.3 to 28.6.

Low Agentic Performance And Robustness τ²-Voice tests long and multi-tool interactions, under severe audio degradation making task completion challenging. However, Riva outperforms the same-backbone cascade across all three domains, and is the only open FD-SLMs to demonstrate long-horizon tool-use capability. Regarding robustness, across three evaluation runs, the standard deviation is just 2–3 percentage points in Airline and Retail, and the improvements shown are substantially larger than this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the ACs and all reviewers for their time and constructive feedback, which helped clarify and strengthen our paper.

Reviewers highlighted several strengths of our work:

  1. A practical, well-motivated problem. Reviewers found that unifying interactivity with strong language capabilities and tool use addresses a practical problem (tmHX).
  2. Non-blocking tool stream. Reviewers found integrating tool calls into the backbone's native stream well motivated (tmHX), and noted that it maintains interactivity during tool-calls (vNfR).
  3. Useful datasets. Riva-Dialog and Riva-Entity provide complementary supervision for conversational behavior and accurate spoken tool arguments, with substantial gains in data ablations (tmHX).
  4. Efficient decision-window RL. Reviewers appreciated focusing RL on short decision windows rather than full-dialogue rollouts, and combining GRPO with on-policy self-distillation for sparse identifier rewards (vNfR).
  5. Clear presentation, broad evaluation, and open recipe. Reviewers found the paper clearly presented with strong results against open full-duplex baselines (tmHX), and highlighted the detailed open recipe for building full-duplex models (2jG7).

Summary of new results and revisions:

  1. RL ablations (tmHX, 2jG7). Our ablations demonstrate that GRPO improves interactivity on FDB v1, while self-distillation further improves tool calling by providing dense supervision for tool arguments, raising τ²-Voice P@1 from 22.3 to 28.6.

  2. Same-backbone cascade comparison (tmHX, vNfR). Riva outperforms the cascade across all three τ²-Voice domains. On FDB v3, it achieves competitive accuracy (42 vs. 44 P@1) with substantially lower response latency (0.56 vs. 2.96 s), demonstrating that native interactivity and tool use can be combined without a large loss in task accuracy.

  3. Generalization and reliability (tmHX). On held-out VoiceAgentBench, Riva achieves a competitive final pass rate with 3× lower latency. The cascade performs better in tool selection and schema accuracy, but struggles with spoken entity recognition, highlighting the importance of accurate tool arguments to final task success. Across three τ²-Voice runs, standard deviations are 2-3 percentage points in Airline and Retail, with higher variability in Telecom. The ablation gains in Airline and Retail substantially exceed this observed variation.

  4. Contribution clarification (2jG7). We state in the paper and reiterate that we adopt the thinker-talker architecture and treat the micro-turn size as a design choice. Our novelty lies in the non-blocking tool-stream design and our pre-training and post-training recipes and datasets. Our ablations establish their substantial contributions to unifying interactivity, language, and agentic capabilities in an open FD-SLM that outperforms or is competitive with prior open FD-SLMs across six benchmarks.

Apart from these results, we added synthetic-data validation, speech-quality assessments, experiments on interaction during tool calls, and implementation clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.