- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Based on my reading of the introduction, the paper primarily aims to enhance tool-calling capabilities of full-duplex speech language model by leveraging the backbone LLM’s native tool-calling functionality. They also opened the detailed recipe of building FD SLMs.
Critical strengths
- They opened the detailed recipe of building full-duplex speech language models.
Critical weaknesses
- I disagree with the following statement in the abstract: “Existing open models, however, struggle to combine native interaction, in which the model itself decides when to listen, speak, or yield, with the language and agentic capabilities of a strong large language model (LLM).” There are already many full-duplex spoken dialogue models, including Moshi, PersonaPlex, and Raon-speech, that decide when to speak and when to yield.
- Raon-speech is also an open-source full-duplex speech language model that adopts a thinker–talker architecture with a lightweight talker. The novelty relative to this model is unclear.
- Using the backbone’s native tool-calling functionality appears to be the paper’s main novelty, but this seems somewhat weak.
- The explanation of tool-calling data construction appears insufficient, given that enhancing tool-calling capabilities seems to be the paper’s main contribution.
- In particular, the description of Riva-Dialog in lines 275–293 does not sufficiently explain the closed set of tools or the format used to call them. These details should be provided in the main text or described thoroughly in the appendix with a corresponding reference.
- The description of GRPO in lines 303–320 refers to “short decision windows around the relevant event,” but does not specify exactly how many windows or turns are used for RL.
- In agentic RL, it is necessary to consider whether a particular tool was called within a multi-turn scenario. The credit assignment algorithm is also not clearly described.
- The paper appears to use several techniques, including noise augmentation and on-policy self-distillation, but the presentation feels scattered. Noise augmentation seems more appropriate for the dataset section, and the paper should establish more clearly what its main contributions are.
- The evaluation is substantially insufficient. For example, Table 1 reports only TT Fluency, Instr. Foll., and Task Metric for Full-Duplex Bench v2, whereas the official FDB v2 reports TT and IF across four categories: Daily, Correction, Entity Tracking, and Safety.
- This raises the concern that the table may cherry-pick favorable results. If space is limited, the full evaluation results should at least be included in the appendix.
- Regarding the writing, too many sentences use colons. In lines 70–89, presenting Representation, Data, and Decisions within a single paragraph makes the text difficult to read.
- The LoRA rank is not specified in the training description in lines 249–261.
- In line 290, a space is missing in “markers.5).”
Questions for authors
- What is the novelty of the proposed model relative to Raon-speech, which also adopts a thinker–talker architecture with a lightweight talker?
- The thinker’s 480 ms interval seems relatively long compared with the 80 ms inference intervals of Moshi and PersonaPlex. Is 480 ms really appropriate? Would this not introduce a consistent delay when a person converses with the model?
- What LoRA rank was used for the training described in lines 249–261?
- For Riva-Dialog in lines 275–293, what closed set of tools was defined, and in what format were these tools called? Please provide these details or add a reference to a detailed explanation in the appendix.
- In lines 303–320, exactly how many windows or turns are used for RL when applying GRPO to “short decision windows around the relevant event”?
- How does the RL setup account for whether a particular tool is called within a multi-turn scenario, and what credit assignment algorithm is used?
- Please consult and cite the following relevant work:
- HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models. (https://arxiv.org/abs/2610.07727)
- Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models. (https://arxiv.org/abs/2606.11167)
- Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning. (https://arxiv.org/abs/2607.07148)
- Could you provide the full FDB v2 evaluation results, including TT and IF for all four categories, at least in the appendix?
- I would also like to see results on FDB v1 to assess the model’s full-duplex turn-taking and pause-handling capabilities.
- Could Representation, Data, and Decisions in lines 70–89 be organized into bullet points rather than presented within a single paragraph?
- Please add a space between “markers.” and “5)” in line 290.
I reviewed the paper without using AI, so there may be points I have missed. I will raise my score if the authors provide sufficient and convincing explanations. Thank you.
4 comments
Response to Reviewer 2jG7
Additional Results On Full-Duplex Bench
As requested, we report the FDB v1 results and per-category FDB v2 breakdown below. Riva demonstrates competitive turn-taking and substantially better pause and interruption handling. Here Syn/CANDOR represent pause handling.
| Method | Syn. TOR↓ | CANDOR TOR↓ | BC TOR↓ | BC Freq.↑ | BC JSD↓ | TT TOR↑ | TT Lat.↓ | Int. TOR↑ | Int. Score↑ | Int. Lat.↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| Moshi | 1.000 | 1.000 | 0.491 | 0.179 | 0.684 | 0.714 | 0.27 | 0.910 | 3.32 | 0.97 |
| Raon | 0.949 | 0.861 | 0.545 | 0.144 | 0.714 | 0.899 | 0.24 | 0.990 | 3.01 | 0.21 |
| MiniCPM-o 4.5 | 0.248 | 0.366 | 0.610 | 0.013 | 0.930 | 0.840 | 0.54 | 0.925 | 4.37 | 0.73 |
| Ours | 0.022 | 0.042 | 0.000 | 0.032 | 0.839 | 1.000 | 0.60 | 1.000 | 4.56 | 0.59 |
| Domain | Turn-taking ↑ | Instr. Following ↑ | Task ↑ |
|---|---|---|---|
| Daily | 4.40 | 3.95 | 3.60 |
| Correction | 3.42 | 3.48 | 4.40 |
| Entity tracking | 3.98 | 4.06 | 3.90 |
| Safety | 4.28 | 4.56 | 4.90 |
Additional Details On RL & References
Given the preceding context of a dialogue, we roll out only the model’s next one turn around a specific event, such as an interruption, a tool call, or a turn transition, typically 4–8 s. The rewards are computed within this window, and since each reward applies directly to the single decision that produced it (such as a tool-call or turn-taking), we do not need a multi-turn credit assignment algorithm.
Moreover, we now reference the three papers shared, in Sec 3.2.
Tool Format And Environments
We use the Hermes tool-call format used by Qwen3, and now describe this Appendix A.1 The full set of the environments and the tools we use are now presented in Table 6 in the Appendix C.2.
Low Rank Implementation Details
We used a LoRa rank of 64, and did not find sufficient difference in 32/64/128 in initial preliminary experiments. We have added this information in Training Details.
[1] Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction
Thank you for the detailed response and clarifications. While several of my concerns regarding evaluation and implementation details have been addressed, my main concern about novelty remains. In particular, the architectural novelty over Raon-Speech appears limited, the 480 ms micro-turn seems more like a design choice than a methodological contribution, and a substantial portion of the contribution lies in the pre-training/post-training recipe and dataset construction. Therefore, I will maintain my current score.
Response To Reviewer 2jG7
We thank the reviewer for responding and are glad that several concerns regarding evaluation and implementation have been addressed.
We would like to clarify that we never claimed the thinker-talker architecture as a contribution. Our paper states that we adopt it, and the 480 ms micro-turn size is a design choice.
As noted by Reviewers tmHX and vNfR, our key architectural contribution is the non-blocking tool stream, which maintains full-duplex interaction during tool-call and execution.
We agree with the reviewer that a substantial portion of the contribution is additionally in pre-training, post-training recipe and training datasets, which contribute to our goal:
-
Pre-training recipe: which narrows the speech-text modality gap under streaming input, preserving the backbone's language capabilities, together with Riva-Entity dataset, which improves spoken entity recognition, a prerequisite for accurate tool calls.
-
Post-training recipe: supervised fine-tuning on Riva-Dialog dataset, which jointly supervises general assistant skills and full-duplex interaction, followed by RL over short decision windows that refines interactivity and tool calling.
Our component ablations establish the benefits of these recipes and datasets. Together, these unify native interactivity, strong language capabilities, and tool calling in a single FD-SLM, which outperforms or is competitive with prior open FD-SLMs across six benchmarks.
Response to Reviewer 2jG7
We thank the reviewer for the detailed assessment. We understand that you did not find the paper especially informative, as reflected in your rating and we acknowledge that our presentation can be improved to convey our contributions better. The main goal of Riva is to have a single full-duplex speech language model (FD-SLM) that unifies interaction, language capabilities and native agentic capabilities. We show that achieving this goal requires contributions across a non-blocking tool-stream design, pre-training and post-training recipe and new training datasets across both. We have addressed your comments, and added experimental results below, and we would greatly appreciate any further candid feedback.
Improved Presentation
Thank you for this suggestion. We have restructured the abstract, and the introduction to convey our contributions more clearly, across separate paragraphs. We have also moved the noise augmentation to the dataset section.
Clarifying The Abstract Claim
We thank you for pointing this out, and we agree that existing models, including Moshi, PersonaPlex, and Raon-Speech, support native turn-taking. However, they exhibit substantially weaker language capabilities and lack native agentic capabilities. Our main claim was that Riva “combines” native interaction, strong language (intelligence) and agentic capabilities. Which to the best of our knowledge, no other open-source model does. We have revised the abstract to clarify this.
Novelty Of Native Tool-Calling
The challenge is not the tool-call format itself, but making tool calls work inside a full-duplex stream. A complete tool call often takes 2–3 s to generate, during which a naively fine-tuned FD-SLM cannot listen or respond, blocking interaction. Our tool stream design splits calls across micro-turns and interleaves them with the user and assistant streams, so the conversation continues while calls are generated, executed, and their results consumed.
Beyond the stream design, to our knowledge no existing datasets target argument errors from spoken entities, or provide full-duplex tool-use conversations with features like user repairs, disfluencies, and interactive spoken fillers.
Novelty Relative to Raon-Speech
We agree that thinker-talker architectures exist, and in fact our paper states that we “adopt the thinker–talker design”. Our main contributions are a novel tool stream that enables non-blocking, streaming tool calls; a pre-training recipe that narrows the gap to the backbone’s intelligence; post-training recipe that enables conversational capabilities and new training datasets across these stages. Together these lead to a highly capable model; Riva substantially outperforms Raon-SpeechChat on most benchmarks while adding native tool-calling capabilities that it lacks.
Justification For Micro-Turn Size
Prior work such as MiniCPM-o 4.5 [1] shows that temporal granularity has a capacity-latency tradeoff, which we also encounter in our preliminary experiments. Therefore, we choose 480ms as our micro-turn size to balance general capability and latency. Importantly, the inference implementation also substantially affects user perceived latency which we further optimize to improve interactivity (Appendix A.4)