- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces HiPLEX, a reinforcement learning framework for full-duplex speech language models that aims to jointly optimize speaking timing and response content. It separates the text policy into a control policy and a content policy to determine when to speak and what to say. Event-causal credit masks are used to assign timing rewards to relevant control decisions and semantic rewards to content generation. Experiments on Moshi and PersonaPlex show that HiPLEX improves several speaking timing metrics and makes the models' conversational timing more closely resemble human behavior.
Critical strengths
The paper addresses an important and underexplored problem in full-duplex speech interaction, namely how to balance speaking timing and response content. The experimental evaluation is also relatively comprehensive, covering multiple interaction scenarios, two model families, ablation studies, and comparisons with existing approaches, providing a broad assessment of the proposed method.
Critical weaknesses
1.The overall advantage over existing RL approaches remains unclear. Compared with the reference results in Table 1, HiPLEX performs better on some metrics but worse on others, particularly on PersonaPlex. These mixed results make it difficult to establish a clear overall advantage over existing RL approaches.
2.The evaluation focuses mainly on turn-taking metrics and response quality in specific interaction scenarios. Since HiPLEX also optimizes semantic content, evaluating the models on general audio-language benchmarks such as OpenAudioBench would help determine whether the method preserves broader audio understanding and question-answering capabilities beyond the targeted scenarios.
Questions for authors
Could the authors evaluate HiPLEX on OpenAudioBench to verify that its general reasoning and knowledge capabilities do not degrade compared with the baseline?
8 comments
(continued) Entries are means across three training seeds; Appendix K, Table 12 provides the corresponding variability. Pause and backchannel restraint improve in every paired seed. These results demonstrate reproducible interaction improvements while maintaining similar mean judged response quality. The semantic scores vary more across HiPLEX seeds, so we distinguish these replicated timing improvements from the higher semantic score of the selected checkpoint in Table 1.
PersonaPlex’s training history and additional controlled comparison. PersonaPlex is itself further trained from Moshi, with additional voice and role conditioning. The two models therefore share a backbone but enter RL post-training with different conversational behaviors and training histories. As documented in Appendix G, PersonaPlex requires a different learning rate and reward balance to maintain responsiveness. Its evaluation consequently tests HiPLEX’s applicability across different post-training histories within the same model family.
To distinguish HiPLEX’s benefits from those of reward rebalancing, we additionally trained GRPO using the same rebalanced rewards as HiPLEX:
| Model | CANDOR pause takeover ↓ | Backchannel takeover ↓ | Smooth-turn response rate ↑ | Interruption response rate ↑ | Interruption Judge ↑ | Interruption latency ↓ |
|---|---|---|---|---|---|---|
| PersonaPlex + rebalanced GRPO | 0.606 | 0.018 | 0.723 | 0.955 | 2.150 | 0.583 s |
| PersonaPlex + HiPLEX | 0.218 | 0.018 | 0.714 | 0.975 | 2.938 | 0.590 s |
With the same reward settings, HiPLEX combines stronger natural-pause restraint with higher interruption response rate and judged quality while maintaining the same backchannel takeover rate. The higher quality score is accompanied by more qualifying interruption responses—195/200 versus 191/200—rather than fewer. The remaining trade-offs are a 0.009 lower smooth-turn response rate and 7 ms longer interruption latency.
We also extended the post-boundary diagnostic discussed in Appendix J to distinguish newly initiated speech from continuation. HiPLEX initiates new post-boundary speech in 47/119 clips, compared with 28/119 for rebalanced GRPO, an increase of 16.0 percentage points. Thus, stronger restraint is accompanied by more frequent new responses after the user finishes in this comparison. This diagnostic identifies response initiation; contextual appropriateness is assessed separately through listening.
Complementary evidence across all four interaction axes. Within the limited rebuttal period, we conducted a blinded, paired listening study with 15 independent evaluators, assessing timing and content on the same exchanges. All sampled exchanges were retained regardless of response timing or presence. Evaluators separately flagged cases where a response was required but absent; neither HiPLEX nor GRPO had such cases in this study:
| Model | Overall timing ↑ | Overall content ↑ |
|---|---|---|
| Moshi + original GRPO | 1.99 | 2.60 |
| Moshi + HiPLEX | 3.71 | 3.72 |
| PersonaPlex + rebalanced GRPO | 2.93 | 2.69 |
| PersonaPlex + HiPLEX | 3.96 | 3.54 |
Scores use a 1–5 scale and are computed by first averaging each evaluator’s ratings over the relevant assigned examples, then averaging these evaluator-level means. Timing ratings favor HiPLEX across all four interaction categories on both Moshi and PersonaPlex. Overall content ratings also favor HiPLEX on both models. In particular, timing and content ratings both improve in turn taking and interruption handling, supporting useful participation in situations that require active responses. The independent listening results show that HiPLEX provides a better perceived conversational experience than the corresponding GRPO baseline on both models. This conclusion describes the evaluated exchanges and checkpoints; it does not imply statistical significance or superiority on every benchmark metric.
(continued)
Clarifying the reference results and the scope of the overall advantage. The reference results in Table 1 are numbers reported by Ohashi et al. (2026), Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models. Although that work describes training and evaluation choices and releases model checkpoints, its RL training implementation and complete evaluation code/configuration were not available to us. We therefore retain these published numbers as reference results: they provide useful context, but we cannot establish identical experimental conditions or attribute cross-paper score differences solely to the learning method. Our controlled evidence comes from GRPO and HiPLEX evaluated in our own setup, including the additional reward-matched PersonaPlex comparison and independent paired listening study. Our original PersonaPlex comparison still retains a lower smooth-turn response rate for HiPLEX, 0.714 versus 0.882, and longer interruption latency, 0.590 versus 0.548 s. We will make this distinction explicit in Table 1 and the results discussion.
To make the intended interpretation explicit, we will add the following statement to the main text:
“HiPLEX’s benefits should be assessed jointly across the four interaction axes. On Moshi, improved pause and backchannel restraint replicate across three training seeds, alongside lower mean interruption latency and similar mean judged response quality. On PersonaPlex, stronger restraint and improved interruption response rate and quality coexist with trade-offs in smooth-turn response rate and interruption latency. Independent, paired human evaluations show that HiPLEX provides a better perceived conversational experience than the corresponding GRPO baseline on both models, with higher overall timing and content ratings. The human-rating gains are smaller on PersonaPlex than on Moshi, qualitatively consistent with the more mixed Full-Duplex-Bench results on PersonaPlex. The magnitude and balance of the gains depend on the model’s post-training history and reward configuration.”
Thank you for emphasizing the importance of evaluating broader audio-language capabilities beyond the targeted interaction scenarios.
W1. Overall advantage over existing RL approaches
The improvements on PersonaPlex are less pronounced than those on Moshi. Nevertheless, Full-Duplex-Bench evaluates four complementary interaction axes—pause handling, backchanneling, turn taking, and interruption handling—and HiPLEX’s practical benefit should be assessed through the balance across these behaviors. On PersonaPlex, HiPLEX substantially reduces unwanted takeovers during pauses and backchannel opportunities while improving interruption response rate and judged response quality relative to our original GRPO baseline. These gains coexist with a lower smooth-turn response rate and slightly longer interruption latency. The independent human evaluation shows that HiPLEX provides a better perceived conversational experience than GRPO on both models, with higher overall timing and content ratings. These perceptual gains accompany the reported trade-offs in individual benchmark metrics. We apologize for not explaining this balance clearly enough in the manuscript and will revise the main results discussion to make the benefits and remaining trade-offs explicit.
Methodological novelty and its practical value. HiPLEX contributes a structured credit-assignment mechanism for jointly optimizing conversational timing and semantic content. As discussed in Appendix A, prior approaches already distinguish timing from content. HiPLEX advances this direction by exactly factorizing the existing text policy into a per-frame control decision over PAD/EPAD/CONTENT and a conditional content-token decision, without adding parameters. Timing advantages update the control factor, while semantic advantages update the conditional content factor.
Crucially, the event-causal masks use generated speech episodes and the advantage sign to identify the decisions that receive timing feedback: a delayed response penalizes the preceding waiting decisions, whereas a timely response credits its onset. Appendices C and E detail the factorization and masking mechanism. The novelty therefore lies in which decisions receive which feedback, beyond simply combining timing and semantic rewards.
The ablations connect this mechanism to the observed benefits. In Table 4, holding the factorization and rewards fixed, all-frame credit produces a turn-response rate of 0.958 and interruption latency of 0.874 s, whereas event-causal credit achieves 1.000 and 0.439 s. Random masking with matched sparsity reduces the turn-response rate to 0.067. Thus, neither factorization alone nor sparse updates explains the observed balance. Table 3 further shows that adding semantic feedback to HiPLEX improves judged response quality from 2.332 to 4.083 while also improving natural-pause restraint, backchannel restraint, and interruption latency. This demonstrates the practical value of jointly optimizing timing and content through structured credit assignment.
Replicated advantages on Moshi. Across three training seeds, HiPLEX improves multiple interaction dimensions relative to GRPO: