acceptodds
Under review as a conference paper at ICLR 2027

HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models

13 views

Abstract

As human–AI interactions become more conversational, full-duplex speech language models capable of natural real-time dialogue are growing in importance. Beyond generating appropriate responses, these models must coordinate turn-taking, backchanneling, and floor management in real time. Reinforcement learning (RL) provides a way to refine these behaviors through direct feedback on interaction outcomes. However, existing RL methods either apply timing feedback to a token policy or optimize semantic content, leaving the joint improvement of timing and content unresolved. We introduce HiPLEX, an RL framework that factorizes a pretrained full-duplex text policy into a control policy that decides when to emit content and a conditional content policy that decides what to emit. The first factor selects among 'pad', 'epad', and 'con'. The second selects a token only when 'con' is chosen. This hierarchy describes conditional actions within each frame and uses the model's existing text head. We route timing advantages to the token-group factor through event-causal masks derived from generated speech episodes, and route an LLM-judge semantic advantage to the conditional content factor. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and backchannel opportunities, and shortens post-interruption response latency relative to GRPO, while maintaining comparable judged interruption-response quality. On Moshi and PersonaPlex, HiPLEX better matches pooled human turn-timing and backchannel-rate marginals than GRPO.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-duplex system, Speech language model, Reinforcement learning, Hierarchical credit assignment

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_uwqEOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces HiPLEX, a reinforcement learning framework for full-duplex speech language models that aims to jointly optimize speaking timing and response content. It separates the text policy into a control policy and a content policy to determine when to speak and what to say. Event-causal credit masks are used to assign timing rewards to relevant control decisions and semantic rewards to content generation. Experiments on Moshi and PersonaPlex show that HiPLEX improves several speaking timing metrics and makes the models' conversational timing more closely resemble human behavior.

Critical strengths

The paper addresses an important and underexplored problem in full-duplex speech interaction, namely how to balance speaking timing and response content. The experimental evaluation is also relatively comprehensive, covering multiple interaction scenarios, two model families, ablation studies, and comparisons with existing approaches, providing a broad assessment of the proposed method.

Critical weaknesses

1.The overall advantage over existing RL approaches remains unclear. Compared with the reference results in Table 1, HiPLEX performs better on some metrics but worse on others, particularly on PersonaPlex. These mixed results make it difficult to establish a clear overall advantage over existing RL approaches.

2.The evaluation focuses mainly on turn-taking metrics and response quality in specific interaction scenarios. Since HiPLEX also optimizes semantic content, evaluating the models on general audio-language benchmarks such as OpenAudioBench would help determine whether the method preserves broader audio understanding and question-answering capabilities beyond the targeted scenarios.

Questions for authors

Could the authors evaluate HiPLEX on OpenAudioBench to verify that its general reasoning and knowledge capabilities do not degrade compared with the baseline?

8 comments

W1. Overall advantage over existing RL approaches

The improvements on PersonaPlex are less pronounced than those on Moshi. Nevertheless, Full-Duplex-Bench evaluates four complementary interaction axes—pause handling, backchanneling, turn taking, and interruption handling—and HiPLEX’s practical benefit should be assessed through the balance across these behaviors. On PersonaPlex, HiPLEX substantially reduces unwanted takeovers during pauses and backchannel opportunities while improving interruption response rate and judged response quality relative to our original GRPO baseline. These gains coexist with a lower smooth-turn response rate and slightly longer interruption latency. The independent human evaluation shows that HiPLEX provides a better perceived conversational experience than GRPO on both models, with higher overall timing and content ratings. These perceptual gains accompany the reported trade-offs in individual benchmark metrics. We apologize for not explaining this balance clearly enough in the manuscript and will revise the main results discussion to make the benefits and remaining trade-offs explicit.

Methodological novelty and its practical value. HiPLEX contributes a structured credit-assignment mechanism for jointly optimizing conversational timing and semantic content. As discussed in Appendix A, prior approaches already distinguish timing from content. HiPLEX advances this direction by exactly factorizing the existing text policy into a per-frame control decision over PAD/EPAD/CONTENT and a conditional content-token decision, without adding parameters. Timing advantages update the control factor, while semantic advantages update the conditional content factor.

Crucially, the event-causal masks use generated speech episodes and the advantage sign to identify the decisions that receive timing feedback: a delayed response penalizes the preceding waiting decisions, whereas a timely response credits its onset. Appendices C and E detail the factorization and masking mechanism. The novelty therefore lies in which decisions receive which feedback, beyond simply combining timing and semantic rewards.

The ablations connect this mechanism to the observed benefits. In Table 4, holding the factorization and rewards fixed, all-frame credit produces a turn-response rate of 0.958 and interruption latency of 0.874 s, whereas event-causal credit achieves 1.000 and 0.439 s. Random masking with matched sparsity reduces the turn-response rate to 0.067. Thus, neither factorization alone nor sparse updates explains the observed balance. Table 3 further shows that adding semantic feedback to HiPLEX improves judged response quality from 2.332 to 4.083 while also improving natural-pause restraint, backchannel restraint, and interruption latency. This demonstrates the practical value of jointly optimizing timing and content through structured credit assignment.

Replicated advantages on Moshi. Across three training seeds, HiPLEX improves multiple interaction dimensions relative to GRPO:

ModelCANDOR pause takeover ↓Backchannel takeover ↓Smooth-turn response rate ↑Interruption Judge ↑Interruption latency ↓
Moshi + GRPO0.4950.2970.9923.9270.631 s
Moshi + HiPLEX0.3290.1091.0003.9140.466 s

(continued) Entries are means across three training seeds; Appendix K, Table 12 provides the corresponding variability. Pause and backchannel restraint improve in every paired seed. These results demonstrate reproducible interaction improvements while maintaining similar mean judged response quality. The semantic scores vary more across HiPLEX seeds, so we distinguish these replicated timing improvements from the higher semantic score of the selected checkpoint in Table 1.

PersonaPlex’s training history and additional controlled comparison. PersonaPlex is itself further trained from Moshi, with additional voice and role conditioning. The two models therefore share a backbone but enter RL post-training with different conversational behaviors and training histories. As documented in Appendix G, PersonaPlex requires a different learning rate and reward balance to maintain responsiveness. Its evaluation consequently tests HiPLEX’s applicability across different post-training histories within the same model family.

To distinguish HiPLEX’s benefits from those of reward rebalancing, we additionally trained GRPO using the same rebalanced rewards as HiPLEX:

ModelCANDOR pause takeover ↓Backchannel takeover ↓Smooth-turn response rate ↑Interruption response rate ↑Interruption Judge ↑Interruption latency ↓
PersonaPlex + rebalanced GRPO0.6060.0180.7230.9552.1500.583 s
PersonaPlex + HiPLEX0.2180.0180.7140.9752.9380.590 s

With the same reward settings, HiPLEX combines stronger natural-pause restraint with higher interruption response rate and judged quality while maintaining the same backchannel takeover rate. The higher quality score is accompanied by more qualifying interruption responses—195/200 versus 191/200—rather than fewer. The remaining trade-offs are a 0.009 lower smooth-turn response rate and 7 ms longer interruption latency.

We also extended the post-boundary diagnostic discussed in Appendix J to distinguish newly initiated speech from continuation. HiPLEX initiates new post-boundary speech in 47/119 clips, compared with 28/119 for rebalanced GRPO, an increase of 16.0 percentage points. Thus, stronger restraint is accompanied by more frequent new responses after the user finishes in this comparison. This diagnostic identifies response initiation; contextual appropriateness is assessed separately through listening.

Complementary evidence across all four interaction axes. Within the limited rebuttal period, we conducted a blinded, paired listening study with 15 independent evaluators, assessing timing and content on the same exchanges. All sampled exchanges were retained regardless of response timing or presence. Evaluators separately flagged cases where a response was required but absent; neither HiPLEX nor GRPO had such cases in this study:

ModelOverall timing ↑Overall content ↑
Moshi + original GRPO1.992.60
Moshi + HiPLEX3.713.72
PersonaPlex + rebalanced GRPO2.932.69
PersonaPlex + HiPLEX3.963.54

Scores use a 1–5 scale and are computed by first averaging each evaluator’s ratings over the relevant assigned examples, then averaging these evaluator-level means. Timing ratings favor HiPLEX across all four interaction categories on both Moshi and PersonaPlex. Overall content ratings also favor HiPLEX on both models. In particular, timing and content ratings both improve in turn taking and interruption handling, supporting useful participation in situations that require active responses. The independent listening results show that HiPLEX provides a better perceived conversational experience than the corresponding GRPO baseline on both models. This conclusion describes the evaluated exchanges and checkpoints; it does not imply statistical significance or superiority on every benchmark metric.

(continued)

Clarifying the reference results and the scope of the overall advantage. The reference results in Table 1 are numbers reported by Ohashi et al. (2026), Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models. Although that work describes training and evaluation choices and releases model checkpoints, its RL training implementation and complete evaluation code/configuration were not available to us. We therefore retain these published numbers as reference results: they provide useful context, but we cannot establish identical experimental conditions or attribute cross-paper score differences solely to the learning method. Our controlled evidence comes from GRPO and HiPLEX evaluated in our own setup, including the additional reward-matched PersonaPlex comparison and independent paired listening study. Our original PersonaPlex comparison still retains a lower smooth-turn response rate for HiPLEX, 0.714 versus 0.882, and longer interruption latency, 0.590 versus 0.548 s. We will make this distinction explicit in Table 1 and the results discussion.

To make the intended interpretation explicit, we will add the following statement to the main text:

“HiPLEX’s benefits should be assessed jointly across the four interaction axes. On Moshi, improved pause and backchannel restraint replicate across three training seeds, alongside lower mean interruption latency and similar mean judged response quality. On PersonaPlex, stronger restraint and improved interruption response rate and quality coexist with trade-offs in smooth-turn response rate and interruption latency. Independent, paired human evaluations show that HiPLEX provides a better perceived conversational experience than the corresponding GRPO baseline on both models, with higher overall timing and content ratings. The human-rating gains are smaller on PersonaPlex than on Moshi, qualitatively consistent with the more mixed Full-Duplex-Bench results on PersonaPlex. The magnitude and balance of the gains depend on the model’s post-training history and reward configuration.”

Thank you for emphasizing the importance of evaluating broader audio-language capabilities beyond the targeted interaction scenarios.

Reviewer_gqk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HiPLEX is a reinforcement-learning objective for full-duplex speech language models that must decide when to speak as well as what to say. It factorizes the existing text-token distribution into a per-frame choice among pad, epad and content, followed by a conditional content-token choice when speech is produced. No separate controller or new policy head is added. Timing rewards for turn onset, pauses, backchannels and interruption handling are routed to the first factor through masks derived from the model's generated speech episodes. An ASR transcript and LLM judge provide a semantic reward for the conditional content factor. The authors compare the resulting method with a reproduced GRPO baseline on Moshi and PersonaPlex using Full-Duplex-Bench v1, with ablations for reward routing and semantic feedback.

Critical strengths

The credit-assignment idea is well suited to the streaming setting. A late reply and a timely reply can contain similar tokens near the boundary but deserve different feedback. HiPLEX uses the generated episode to distinguish waiting frames from the actual onset and routes the advantage accordingly. The Moshi ablation is useful here: random masks with matched sparsity yield a turn-response rate of only 0.067, while event-causal routing reaches 1.000 in that table; an all-frame factorized variant loses some turn response and has longer interruption latency. Thus the gain cannot be explained simply by using fewer frames or by changing the token probability factorization.

The paper also avoids hiding the participation problem. Timing-only training lowers judged response quality in Table 3, while the semantic reward improves HiPLEX's judge score from 2.332 to 4.083 and improves several timing measures at the same time. The authors report both conditional answer quality and response-rate measures and show that chasing one can suppress the other. The three-seed Moshi study, with pause takeover 0.329 ± 0.083 for HiPLEX versus 0.495 ± 0.050 for GRPO, is more persuasive than a single favorable checkpoint. The human-marginal comparison is a helpful diagnostic even if it does not measure naturalness directly.

Critical weaknesses
  1. On PersonaPlex, the result is not a clean extension of the Moshi result. HiPLEX reduces pause and backchannel takeover, but smooth-turn response rate drops by 0.168 and interruption latency is worse than GRPO. The authors argue that the official turn metric undercounts useful post-boundary activity, but they also acknowledge that this activity may be speech that began before the user finished. Moreover, the PersonaPlex recipe required a different learning rate and reward rebalancing to avoid a near-silent policy. These facts suggest that the timing/participation tradeoff is sensitive to the starting model and reward configuration, so a broad claim of improved full-duplex behavior across both families is premature.

  2. The semantic-quality evidence is less direct than the timing evidence. It uses ASR and an LLM judge, with the main judge score conditional on producing a qualifying response. That is a reasonable automated signal for training, but it can miss poor responses that are not transcribed accurately and can make a more selective model look better. The judged quality difference is also less stable across Moshi seeds than the restraint metrics. I would like an independent listening evaluation that jointly rates timeliness, interruption handling and content on the same clips. The pooled human marginals are informative but cannot show whether the model behaves appropriately conditional on the particular user utterance; a policy could match each marginal while getting individual exchanges wrong.

Questions for authors
  1. Can the authors provide a paired human listening study on Moshi and PersonaPlex that rates whether the agent responded at the right time and whether its response was useful, including nonresponses and premature responses in the denominator? This would materially test whether lower takeover rates correspond to a better conversation rather than a more hesitant policy.

  2. For PersonaPlex, can the authors separate newly initiated post-boundary turns from words belonging to speech that started before the user finished, and report the same measure for GRPO under matched reward tuning? Please also clarify whether the rebalanced PersonaPlex reward weights were selected on an independent validation set or after examining benchmark outcomes.

10 comments

Thank you for the careful reading and detailed feedback. Your comments prompted additional experiments and helped us clarify the method, its practical benefits, and the scope of our conclusions. We address each weakness and question below, connecting the related evidence where appropriate.

W1. PersonaPlex trade-offs and reward-configuration sensitivity Thank you for highlighting the need to distinguish the PersonaPlex findings from the replicated Moshi results and to control for reward rebalancing. We have added a GRPO experiment using the rebalanced PersonaPlex rewards employed by HiPLEX. We also explicitly separate newly initiated post-boundary turns from speech that began before the user finished, using the same decomposition for HiPLEX and reward-rebalanced GRPO, as reported in Q2.

ModelPause Syn TOR ↓Pause CANDOR TOR ↓BC TOR ↓BC Freq ↑BC JSD ↓Turn TOR ↑Turn latency ↓Int. TOR ↑Int. Judge ↑Int. latency ↓
PersonaPlex + rebalanced GRPO0.8120.6060.0180.1080.7880.7230.079 s0.9552.1500.583 s
PersonaPlex + HiPLEX0.7150.2180.0180.1060.7690.7140.002 s0.9752.9380.590 s

With these shared reward settings, HiPLEX achieves lower takeover on both pause subsets, equal backchannel takeover, and higher interruption response rate and conditional judged quality. The interruption response counts are 195/200 versus 191/200, so the higher judge score is not accompanied by a lower response rate. This does not eliminate transcription or response-selection limitations, but it argues against explaining the quality difference simply through greater selectivity.

Two additional findings help distinguish useful restraint from a merely hesitant policy. First, HiPLEX produces newly initiated post-boundary speech more frequently than this rebalanced baseline, as detailed in Q2. Second, the independent listening evaluation shows higher mean timing and content ratings across all four interaction categories on PersonaPlex. Overall timing increases from 2.93 to 3.96 and overall content from 2.69 to 3.54 against the same rebalanced GRPO baseline, as detailed in Q1. These paired assessments show that HiPLEX provides a better perceived conversational experience on the evaluated PersonaPlex exchanges.

We nevertheless retain the configuration-sensitivity qualification. Against the original GRPO, the reported 0.168 smooth-turn TOR deficit and 42 ms longer interruption latency remain. Figure 4 and Appendix G already document PersonaPlex’s sensitivity to the learning rate and reward balance. We distinguish these configuration-dependent findings from the Moshi restraint improvements replicated across three seeds in Table 12, rather than presenting PersonaPlex as a uniform improvement on every metric.

W2. Semantic quality and context-dependent appropriateness

Thank you for emphasizing the distinction between automated quality scores and appropriate behavior in individual exchanges. We agree that ASR-based, response-conditional judge scores cannot establish conversational quality on their own.

Appendix H distinguishes the training reward model and rubric from the benchmark evaluator. However, this separation does not remove the limitations of transcription or conditioning evaluation on a qualifying response. We therefore complement the automated results with independent, paired listening judgments of timing and content on the same exchanges. The protocol retains all sampled exchanges regardless of response timing or presence and separately checks for failures to respond when a response is required.

We also agree that matching pooled human marginals does not establish context-conditioned appropriateness. Table 2 is a complementary distributional diagnostic, with this limitation explicitly discussed in Appendix N. For example, HiPLEX has lower turn-timing distance than the original GRPO on both Moshi (0.902 vs. 1.781) and PersonaPlex (2.627 vs. 3.511). This diagnostic retains nonparticipation: unanswered turns receive a 6 s ceiling, and backchannel clips with no speech contribute zero events per minute. Nevertheless, these marginal comparisons cannot determine whether a particular response was appropriate. The listening study addresses that separate question.

The expanded human results show higher mean timing ratings for HiPLEX across all four interaction categories on both Moshi and PersonaPlex, together with higher overall content ratings on both models. The largest content gains occur in interruption handling on Moshi (3.67 vs. 2.00) and turn taking on PersonaPlex (3.39 vs. 2.17). Timing and content ratings both improve in turn taking and interruption handling on both models, providing evidence that the benefits extend to situations requiring active responses. Taken together, the independent human ratings show that HiPLEX provides a better perceived conversational experience than GRPO on the evaluated exchanges for both models, combining more appropriate timing with higher overall content ratings.

We retain the limitation concerning semantic-quality robustness across training seeds. Table 12 shows that the Moshi judged-quality differences are less stable than the restraint improvements. The listening study evaluates the selected checkpoints and does not establish semantic superiority across independent training runs.

Q1. Paired listening including nonresponses and premature responses

Thank you for requesting a paired listening study that evaluates timing and usefulness without excluding nonresponses or prematurely initiated responses. Within the limited rebuttal period, we conducted the study and report the full results for each interaction category below.

We constructed a 40-pair evaluation pool by randomly sampling 10 pairs for each of the four interaction axes from the Full-Duplex-Bench evaluation data. Pause and smooth-turn cases use the CANDOR subsets; backchannel and interruption cases use the corresponding benchmark subsets. The pool spans both Moshi and PersonaPlex. We recruited 15 independent evaluators, each assigned 16 pairs (four per interaction axis), with six evaluations collected per pair.

Model identities were hidden, and both presentation order and the assignment of model outputs to A/B were randomized. Participants rated timing and content on a 1–5 scale, with explicit instructions to assess nonresponses and prematurely initiated responses. The timing instructions anchored 1 at severe failures, such as failing to respond when required or already speaking unrelated content prematurely, and 5 at perfectly appropriate timing comparable to human conversation.

Evaluators also used a separate checkbox to identify cases in which the agent should have responded but did not. No such response failures were recorded for either HiPLEX or GRPO in the evaluated clips. All sampled exchanges remained in the evaluation, so the reported ratings were not conditioned on producing a qualifying response or on avoiding premature speech.

For each reported score, we first computed each evaluator’s mean rating over the relevant assigned examples and then averaged these evaluator-level means. Thus, the tables report averages of evaluator-level means, rather than a single pooled mean of all individual ratings.

The comparator is the original GRPO for Moshi and the reward-rebalanced GRPO for PersonaPlex.

Human timing ratings

ModelOverall ↑Turn taking ↑Pause handling ↑Interruption ↑Backchanneling ↑
Moshi + original GRPO1.992.281.611.832.22
Moshi + HiPLEX3.713.563.334.003.94
PersonaPlex + rebalanced GRPO2.932.392.612.724.00
PersonaPlex + HiPLEX3.963.724.223.724.17

Human content ratings

ModelOverall ↑Turn taking ↑Pause handling ↑Interruption ↑Backchanneling ↑
Moshi + original GRPO2.602.612.222.003.56
Moshi + HiPLEX3.724.113.563.673.56
PersonaPlex + rebalanced GRPO2.692.172.722.783.11
PersonaPlex + HiPLEX3.543.393.613.443.72

The independent listening evaluation shows that HiPLEX provides a better perceived conversational experience than the corresponding GRPO baseline on both models. Mean timing ratings are higher in all four interaction categories, with overall scores increasing from 1.99 to 3.71 on Moshi and from 2.93 to 3.96 on PersonaPlex. Overall content scores also increase, from 2.60 to 3.72 on Moshi and from 2.69 to 3.54 on PersonaPlex.

On Moshi, the largest gains occur in interruption handling, where timing improves from 1.83 to 4.00 and content from 2.00 to 3.67. Turn-taking ratings also improve in both timing (2.28 to 3.56) and content (2.61 to 4.11). Together with higher pause-handling and backchannel timing ratings, these results indicate benefits across both restraint and active participation.

On PersonaPlex, HiPLEX receives higher mean timing and content ratings in every category against reward-rebalanced GRPO. The largest timing gain occurs in pause handling (4.22 vs. 2.61), whereas the largest content gain occurs in turn taking (3.39 vs. 2.17). Interruption handling also improves in both timing (3.72 vs. 2.72) and content (3.44 vs. 2.78), showing that the observed benefits extend beyond avoiding unwanted takeovers.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the careful evaluation from reviewers. We have strengthened the paper through additional controlled experiments, independent human evaluation, and broader capability testing, with all manuscript changes highlighted in blue. We summarize the main updates below.

  1. Human evaluation of conversational quality with significance testing (gqk9, uwqE). We conducted a blinded, paired listening study with 15 independent evaluators, 40 pairs, and six assessments per pair, retaining all sampled exchanges regardless of response timing or presence. Overall timing ratings improved from 1.99→3.71 on Moshi and 2.93→3.96 on PersonaPlex. Content ratings improved from 2.60→3.72 and 2.69→3.54, respectively. Statistical tests on the original human ratings found significant improvements in overall timing and content for both models, with p=0.0001p=0.0001 and Holm-adjusted p=0.0004p=0.0004 for each of the four comparisons. Timing ratings also favored HiPLEX across all four interaction categories on both models (§5.4 and Appendix R). We acknowledge the limited sample size and ask that the study’s scale be considered alongside the practical difficulty of recruiting independent evaluators within the short rebuttal period.

  2. Controlled PersonaPlex comparison (gqk9, uwqE). We additionally trained GRPO using the same rebalanced rewards as HiPLEX. Under these matched rewards, HiPLEX reduced CANDOR pause takeover from 0.606→0.218, increased interruption response rate from 0.955→0.975, and improved interruption judge score from 2.150→2.938. The remaining matched-comparison trade-offs are a 0.009 lower smooth-turn response rate and 7 ms longer interruption latency. We explicitly retain the larger trade-offs against the original GRPO and distinguish external reference results from controlled comparisons (§5.1 and Table 1).

  3. New responses versus continued speech (gqk9). We added the requested decomposition of post-boundary activity. HiPLEX initiates new speech after the user finishes in 47/119 clips, compared with 28/119 for reward-matched GRPO, an increase of 16.0 percentage points. Thus, its higher post-boundary activity is not solely explained by continuation of earlier speech. Contextual appropriateness is assessed separately through listening (Appendix J).

  4. Broader capability retention (uwqE). We evaluated six checkpoints on OpenAudioBench. HiPLEX improved Moshi’s speech-to-text score from 27.92→36.94 and speech-to-speech score from 25.14→33.04, exceeding GRPO’s 34.13/30.83. PersonaPlex showed small decreases of 0.41/0.19 points relative to its base model, providing evidence against substantial aggregate degradation on this benchmark (§5.4 and Appendix S).

  5. Reward selection and robustness (gqk9). We clarified that Full-Duplex-Bench initially revealed the failure mode, while subsequent reward selection used a separate Seamless Interaction validation split without further benchmark consultation (§5.2 and Appendix G). We also distinguish the existing three-seed Moshi restraint improvements, with mean pause takeover decreasing from 0.495→0.329 and backchannel takeover from 0.297→0.109, from semantic-quality gains, which remain less stable across seeds (Appendix K and Table 12).

The smaller gains on PersonaPlex also warrant consideration of its training history. PersonaPlex is already fine-tuned from Moshi [1], and sequential fine-tuning can interfere with previously learned capabilities [2]. Prior specialization is a plausible contributor to differing improvement magnitudes, although our experiments do not establish causality. Smaller gains alone do not establish a limitation specific to HiPLEX. The human evaluations show benefits over the corresponding GRPO baselines on both models.

Following these additions, Reviewer uwqE maintained weak acceptance and increased confidence to 4. Reviewer gqk9 acknowledged that the additional experiments addressed part of their concerns while retaining weak rejection. We have made the remaining limitations explicit.

As our submission has received only two reviews, we would particularly appreciate the Area Chair’s independent assessment of the revised manuscript and this additional evidence when forming the final recommendation.

Sincerely,
The Authors

[1] Roy et al. (2026). PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models.

[2] Hui et al. (2025). HFT: Half Fine-Tuning for Large Language Models. ACL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.