acceptodds
Under review as a conference paper at ICLR 2027

Why Full-Duplex Speech Models Collapse in Long Conversations

Abstract

Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
full-duplex speech models, spoken dialogue, long-context, sliding-window attention, KV cache, attention sink, repetition, model collapse, benchmark

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_s4qyOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates the long-horizon generation problem in full-duplex speech LLMs. The authors developed the Full-Duplex Long-Horizon Benchmark (FD-LHB) to evaluate five recent full-duplex models and found that these models tend to collapse during long-horizon inference, such as 30- or 60-minute sessions. The authors then focus on the repetition, silence, and gibberish problems in sliding-window full-duplex models. They propose Initialization-Anchor Pinning (IAP) to address gibberish and silence, and Duplex Repetition Penalty (DRP) to mitigate repetition. The results show that both proposed techniques are highly effective in addressing their respective target problems.

Critical strengths
  • Strong Novelty: This is one of the most interesting papers I have read recently. Most existing full-duplex works focus on model training, developing new model architectures or collecting more data to improve model performance. However, to the best of my knowledge, no existing work has successfully addressed the long-horizon collapse problem in full-duplex models. This paper approaches the problem from a different angle: inference-time intervention. Full-duplex models are known to be highly sensitive to inference parameters, so I personally believe that inference-time manipulation (e.g., through the inference harness) is a promising research direction for this field.
  • Strong Results: The overall results demonstrate that the proposed methods are highly effective. The latency and RTF analyses are also crucial for demonstrating their practical applicability.
  • Good Presentation and Visualization: The paper includes numerous figures and charts that clearly illustrate both the problems and the effectiveness of the proposed methods.
Critical weaknesses
  • Somewhat Limited Backbones: The authors mainly evaluate their proposed methods on Moshi and PersonaPlex, which belong to the same model family (PersonaPlex is a fine-tuned version of Moshi). I understand that there may not be many sliding-window models available, but I wonder whether there are other similar models on which the proposed methods could be evaluated.
  • Combined Results: Some figures and tables aggregate the results from Moshi and PersonaPlex. This makes the results less informative, as it is possible that only one model exhibits the expected improvements while the other does not, and averaging their results could obscure such differences.
Questions for authors
  • I suspect that the first entry in PersonaPlex might correspond to a system prompt or voice prompt token rather than a conversational token. What exactly is preserved in the first KV entry for PersonaPlex?
  • For PersonaPlex, the silence rate only decreases from 85% to 47.5%, which is a less substantial improvement compared to Moshi (from 100% to 7.5%). Have the authors analyzed why the improvement is less pronounced for PersonaPlex?
2 comments

We sincerely thank the reviewer for the very positive assessment, and for recognizing the novelty of our inference-time approach, the strength of the results, and the clarity of the presentation.

W1. Limited backbones

Thank you for the suggestion. We extended our evaluation to MiniCPM-o 4.5, which uses a different architecture from Moshi and PersonaPlex, to further examine the generalizability of our findings. Please see GR1 for details.

W2. Per-model results for Moshi and PersonaPlex

Thank you for the suggestion. We now show separate results for Moshi and PersonaPlex in Appendix B.2 (Figs. 14–17, Tables 9–10). These results show that the main findings hold for both models.

Q1. Contents of PersonaPlex's first KV entry

The first KV entry in PersonaPlex comes from a BOS initialization frame, not a system prompt, voice prompt, or conversation. This frame contains one text BOS token and 16 audio BOS tokens. The voice and text prompts come after this frame. Moshi also starts with the same type of BOS frame. Thus, IAP keeps the initialization state, not the prompt information, in both models.

Q2. Smaller silence reduction in PersonaPlex

Most remaining silences in PersonaPlex are temporary. With IAP+DRP, 19/40 runs still had a two-minute silence, but 18 of them resumed speaking. We have not identified a clear cause, but one possible factor is training data. In its final stage, Moshi was trained only on LLM-written dialogues rendered with TTS, in which it mostly responds to the user and the user rarely pauses mid-utterance. PersonaPlex’s final training also includes real phone conversations (Fisher), which contain natural pauses and listening, so it may stay silent more often.

Thank the authors for their rebuttal. I maintain my positive assessment after reviewing the rebuttal.

Reviewer_4t14OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates whether full-duplex speech models can stay stable over conversations lasting tens of minutes. The paper introduces the Full-Duplex Long- Horizon Benchmark (FD-LHB), which tests five recent models, including Moshi, PersonaPlex, Raon-SpeechChat, BayLing-Duplex, Lychee-FD, on long replayed conversations composed of 40 CANDOR conversations, and in live conversation with Gemini and VITS-TTS. The paper finds that eventually every studied model shows repetition, prolonged silence, gibberish, or context exhaustion. For the two sliding-window models, the paper attributes silence and gibberish to initialization-anchor eviction and attributes repetition to self-reinforcement from recent outputs. The paper then proposes two training-free fixes, that is, initialization-anchor pinning (IAP), which keeps the first KV entry, and duplex repetition penalty (DRP), which is a renormalized CTRL penalty with a PAD-logit safeguard, largely removing the measured failures in replay and adaptive conversations with an LLM partner, at about 0.75 RTF.

Critical strengths
  1. Existing full-duplex benchmarks such as the Full-Duplex-Bench series lack long-horizon evaluation and analysis; yet long-horizon full-duplex conversations (over tens of minutes) are required by many real-world use scenarios. Therefore, the proposed FD-LHB benchmark and model study of this work are valuable to the community. Using FD-LHB, the authors find that every evaluated model eventually either degrades, through repetition, silence, or gibberish, or exhausts its context: the two sliding-window models Moshi and PersonaPlex collapse even though their memory never runs out, and the growing-context models Raon-SpeechChat, BayLing-Duplex, and Lychee-FD can collapse before reaching their context limit. These findings are interesting and will inspire deeper analysis into the causes and fixes.

  2. The diagnostic experiments are well designed for verifying the hypotheses. The analysis in Figure 4 shows collapse onset time versus attention-window length, and the analysis in Figure 5 shows which cache entry and how many entries must be retained to prevent gibberish. Both analyses help verify the hypothesis that the first KV entry alone prevents gibberish. Figure 6 and analysis are useful to verify the hypothesis that recent exposure makes a word recur. Moreover, it is good to see the discussion of the pinned anchor acting as an attention sink.

  3. The proposed fixes are training-free, efficient, and theoretically applicable to Moshi-like full-duplex SDMs.

Critical weaknesses
  1. This is my most critical concern. The study of the five recent full-duplex SDMs in this work, covering sliding-window and context-growing categories, is incomplete. First, PersonaPlex is built on Moshi architecture, so these two sliding-window models are basically with the same architecture. Second, as far as I checked, the paper focused on hypothesizing and verifying the causes of collapse, that is, Initialization-Anchor Eviction and Self-Reinforcing Repetition, for the two sliding-window models Moshi and PersonaPlex in Section 4, and evaluating the proposed two training-free methods IAP and DRP again on the two sliding-window models in Section 6. However, the paper does not analyze the causes of collapse for the three context-growing models, nor provide fixes to mitigate the problems. Therefore, the cause analysis and fixes in this work are only for a single model family. This raises serious concerns on scope and generality of the findings and mitigation approaches.

  2. There are concerns on the impact of IAP and DRP and their interactions and sufficiency of the evaluation metrics. Figure 9a shows that IAP-only improves repetition-free from 35% to 90%, so it seems that most repetitions in Moshi are due to anchor eviction rather than independent self-reinforcement. Figure 9c and Table 12 show that DRP alone achieves 100% repetition-free but also suffers from gibberish at 40/40; however, these results are not sufficiently explained/discussed, it’s unclear whether the failure is fixed. Appendix D.7 also shows that in the A/B study, listeners prefer IAP-alone early in the session, but prefer IAP+DRP in the middle and late phases. This suggests DRP creates a tradeoff, but how to address this tradeoff is not further discussed.

  3. It is unclear how the interventions impact the overall full-duplex capabilities. (a) Based on the paper, DRP penalizes every word in the last 512 lexical tokens, including backchannels, which are central to full-duplex interaction. (b) Also, lowering the PAD logit after 30s of silence may increase barge-in. There are no evaluations w/ and w/o IAP+DRP on full-duplex behaviors as turn-taking, backchanneling (e.g., on Full-Duplex-Bench series), and also response latency.

  4. There are also concerns on the proposed benchmark. The adaptive track of the benchmark involves conversing with Gemini to generate turn-by-turn text then VITS synthesizing them into speech. It is unclear how the adaptive track generates full-duplex interactions since this mechanism suggests half-duplex conversations without interruptions and overlapping speech.

  5. Although it is appealing that IAP and DRP are training-free, it would be useful to compare the performance of the training-free approach with some lightweight fine-tuning methods on long sequences, to enhance the understanding of the positioning of the training-free methods.

Questions for authors
  1. Please address all points under Weaknesses.

  2. What happens with IAP alone but without the first-frame special tokens? Does the first-entry-sufficiency claim still hold?

  3. As shown in Table 10, for Moshi in the adaptive track, IAP+DRP still reaches gibberish in 2/40 sessions. This result needs to be explained.

4 comments

We thank the reviewer for the detailed review, and for finding our long-conversation benchmark useful, our extensive experiments on why models collapse well designed, and our fixes efficient without retraining.

W1-1. Shared architecture

We agree that PersonaPlex is built on Moshi. To our knowledge, MiniCPM-o 4.5 is the only other publicly available full-duplex speech model that supports sliding-window caching, and it uses a different backbone (Qwen3-8B). We found the same collapse pattern in this model and applied IAP and DRP to it. Please see GR1 for details.

W1-2. No diagnosis or fix for growing-context models

Our original focus was on sliding-window models because growing-context models eventually run out of context in long conversations. Nevertheless, we analyzed BayLing-Duplex, a growing-context model, and converted it to sliding-window inference with IAP and DRP to test whether our method also applies. See GR2 for details.

W2. Interaction between IAP and DRP

Thank you for raising these points. We clarify the complementary roles of IAP and DRP.

W2-1. IAP also reduces repetition

As the reviewer noted, IAP also reduces repetition (26/40 → 4/40 in Moshi; Table 14). Repetition in the baseline took two forms: repeating whole sentences, and repeating words without meaning (e.g., “a a an a thank a a”). The repetition that IAP removed was mostly the second form, which is both repetitive and meaningless and therefore meets gibberish and repetition criteria. By removing gibberish, IAP likely removed most of this repetition as well.

However, IAP alone does not fully eliminate repetition. In PersonaPlex, IAP alone removed gibberish, but repetition still occurred in 34/40 runs in our 20-minute ablation; it fell to 0/40 only when DRP was added (Table 15). These results suggest that anchor eviction contributes to repetition mainly through gibberish, while self-reinforcement drives the remaining repetition. Their relative importance differs across models.

W2-2. DRP alone: no repetition but 40/40 gibberish

As the reviewer notes, DRP does not resolve gibberish. Both Baseline and DRP alone exhibit gibberish in 40/40 runs, with similar onset times (13.3 vs. 13.7 minutes). IAP eliminates gibberish (0/40), and combining IAP with DRP eliminates both repetition and gibberish (0/40 each). These results demonstrate the complementary roles of IAP and DRP.

W2-3. Early-phase trade-off of DRP

We acknowledge this trade-off. Our human listening study (Appendix D.8, Table 18) suggests that DRP may suppress natural word reuse before repetition becomes problematic. However, as conversations become longer, repetition becomes a more critical problem, making DRP increasingly important for maintaining coherent speech.

A possible improvement would be to activate DRP adaptively when repetition begins to increase, which we leave for future work. We also acknowledge this trade-off in our limitations (Appendix H.2).

W3. Impact on full-duplex behavior

We evaluated Moshi and PersonaPlex on Full-Duplex-Bench v1 (727 inputs), which tests turn-taking, user interruptions, pauses, and backchannels (Appendix G, Table 21). At the start of a conversation, no cache eviction has occurred, so IAP is identical to the original model, and this comparison shows the effect of DRP alone.

MetricMoshi → +DRPPersonaPlex → +DRP
Smooth turn-taking TOR ↑0.731 → 0.7310.950 → 0.950
User interruption TOR ↑0.905 → 0.8850.915 → 0.935
Pause TOR (CANDOR) ↓0.449 → 0.4540.431 → 0.356
Backchannel frequency (/s) ↑0.103 → 0.0970.097 → 0.096
Backchannel TOR ↓0.473 → 0.7270.273 → 0.255

DRP left smooth turn-taking unchanged and changed the user interruption TOR and backchannel frequency by at most 0.02 in either model. The main differences were in unwanted takeovers: DRP reduced them during user pauses in PersonaPlex (0.431 → 0.356) but increased them during backchannels in Moshi (0.473 → 0.727).

We also fed 5–30 minutes of silent input before the benchmark to see how this behavior changes over a long conversation (Appendix G). Without our method, both models no longer took turns properly after eviction: at 20 minutes, the smooth turn-taking rate fell to 0.050 in Moshi and 0 in PersonaPlex. With IAP+DRP, both models kept taking turns (0.857 and 0.773) and responding to user interruptions (0.995 and 0.945).

W4. Half-duplex nature of the adaptive track

As the reviewer noted, the adaptive track (FD-LHB-LLM) is turn-based. The LLM partner adapts each reply to the model’s output, but it does not interrupt or backchannel while the model is speaking. This track therefore tests long-horizon stability as the dialogue content changes across turns.

To complement this, we also include the CANDOR replay track (FD-LHB-CANDOR), which preserves the natural timing of human conversations, including overlapping speech and interruptions. Although the recorded speech does not adapt to the model, it provides more realistic conversational timing.

Together, the two tracks evaluate long-horizon stability under both content-adaptive dialogue and natural conversational timing.

W5. Comparison with lightweight fine-tuning

We also tested whether lightweight fine-tuning can fix these failures by fine-tuning PersonaPlex with LoRA. All conditions are evaluated on the first 20 minutes of the same 40 CANDOR replays.

MethodTraining dataWindow (train / test)Repetition ↓Prolonged silence ↓Gibberish ↓
Baseline–– / 4 min32/4034/4040/40
IAP+DRP (ours)–– / 4 min0/405/400/40
LoRA20 h4 / 4 min20/4039/407/40
LoRA20 h20 / 20 min29/4040/4015/40
LoRA100 h20 / 20 min40/401/4040/40

LoRA improved some failure types but did not fix all three at once in any tested setting. In contrast, our method reduced all three failures (0/40 repetition, 5/40 prolonged silence, and 0/40 gibberish) without any training. This shows the key advantage of our approach: it improves long-horizon stability at inference time, with no additional training.

Q2. IAP without first-frame special tokens

Both models start from a BOS frame, which IAP keeps as the first entry. When we replaced it with ordinary tokens, all runs failed from the start (Moshi: 13/13; PersonaPlex: 15/15), likely because both models are trained on sequences that start with this frame. Our first-entry claim therefore applies to this standard BOS initialization.

Q3. Remaining gibberish in Moshi adaptive sessions (2/40)

As you pointed out, gibberish remained in 2 of the 40 Moshi sessions, and we inspected both. Moshi responded normally until late in the 20-minute evaluation, and gibberish began at minute 17 in one session and minute 18 in the other session. Compared with the clean 38 sessions, we found no clear warning signs, so the cause remains unclear. However, IAP+DRP reduced gibberish from 40/40 to 2/40 sessions, a 95% reduction. Our goal is to diagnose and address the main causes of long-horizon collapse without training, not to guarantee failure-free generation.

I greatly appreciate the authors' detailed response, which has addressed most of my concerns and questions. I went through the updated manuscript, going through the updated sections in Appendix B, D, E, F, G, and H. The added evaluations and analysis on MiniCPM-o 4.5 and BayLing-Duplex are important to enhance generality of the findings and the proposed methods. The added separate diagnostics for Moshi and PersonaPlex, hyperparameter sensitivity analysis, and also evaluations of full-duplex behaviors are also useful. The clarifications of interactions between DRP and IAP, DRP tradeoff, and comparisons with LoRA are also helpful to address my questions. Therefore, I increased my rating to Weak acceptance.

When checking the FDB-v1 results from Moshi and PersonaPlex w/o and w/ DRP, I noticed that on Moshi+DRP, Backchannel TOR substantially increased from 0.473 to 0.727, while DRP's impact on other full-duplex behaviors of Moshi and PersonaPlex seem to be within reasonable scope. Have you looked into this experiment and could you provide some insights on this large change? Thank you.

Yes, we looked into this and added the analysis to Appendix G ("Backchannel takeovers in Moshi"). We found three things: (1) DRP causes some extra takeovers; (2) however, most newly flagged items are short acknowledgments that the scorer counts as takeovers based on duration and word count; and (3) DRP makes Moshi's backchannels more varied, replacing a standalone "yeah" with responses such as "I see.", though longer ones like "That makes sense." are also counted as takeovers. This reflects the early-conversation cost of DRP noted in our limitations (Appendix H.2). Thank you again for raising your rating and for this careful question.

Reviewer_2re1OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducts a systematic investigation on how full-duplex speech models hold up over long conversations. It introduces FD-LHB, a long-form full-duplex benchmark. Then, five recent models are evaluated, and all of them eventually either collapse into repetition, silence, or gibberish. For sliding-window models, the authors trace gibberish and silence to the eviction of the first KV entry (Initialization-Anchor Eviction), and repetition to self-reinforcement from the model's own recent outputs. They then propose two training-free fixes, Initialization-Anchor Pinning (IAP) and Duplex Repetition Penalty (DRP).

Critical strengths
  1. The paper is well written and easy to follow.
  2. The paper studies failure modes in long-horizon full-duplex conversation, a practical problem that has received little attention, and offers a simple, training-free mitigation for sliding-window models.
Critical weaknesses
  1. Growing-context models such as BayLing-Duplex also suffer from repetition, silence, and gibberish before they reach their context limit. However, the paper does not offer any insightful analysis or discussion of why these models collapse. Because the diagnosis and both fixes are limited to sliding-window models, the paper's broader claim about why full-duplex models collapse in long conversations is only partly supported.
  2. It is unclear how far the mechanism analysis generalizes. The two sliding-window models studied, Moshi and PersonaPlex, share essentially the same architecture and training recipe, so Initialization-Anchor Eviction may be specific to this model family rather than a general property of sliding-window full-duplex models. Perhaps the authors should try MiniCPM-o.
Questions for authors

I don't quite understand the evaluation metrics for Repetition, Silence, and Gibberish, or whether Seq-Rep-4 and Gemini-2.5-Flash use audio or text input.
Additionally, how “spoken words” are defined.

1 comment

We greatly thank the reviewer for the positive evaluation, and for recognizing the clarity of the paper and the practical importance of the problem.

W1. Growing-context model analysis

We agree that our diagnosis covers only sliding-window models. To address this, we analyzed BayLing-Duplex, a growing-context model, to examine why it collapses and whether our method can be applied to it. See GR2 for details.

W2. Generality beyond the Moshi family

Following your suggestion, we added MiniCPM-o 4.5, which uses a different backbone from Moshi. It showed the same collapse pattern as Moshi and PersonaPlex, and we also applied IAP and DRP to it. Please see GR1 for details.

Q1. Definition of evaluation metrics

In each 32-word window, Seq-Rep-4 is the fraction of word 4-grams that repeat an earlier 4-gram in that window. We flag repetition when this fraction reaches 0.30 in two consecutive windows. We measure three failure types using the model’s text output (Appendix A.2): Repetition: Seq-Rep-4 ≥ 0.30 in two consecutive 32-word windows. Silence: Fewer than 3 words per minute for two consecutive minutes. Gibberish: Gemini-2.5-Flash identifies largely uninterpretable text in a one-minute speaking segment.

Q2. Text or audio? What counts as “spoken words”?

All three metrics use the model’s text output, not audio. “Spoken words” are the words in this text, excluding non-lexical control tokens such as PAD and EPAD. We also validated the metrics against audio (Appendices A.3–A.4), which shows that the text-based metrics agree closely with the audio.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response 1

We thank all reviewers for their constructive feedback. We first address the two concerns shared across reviewers, then summarize the revisions.

GR1. Generality beyond the Moshi family

We agree that Moshi and PersonaPlex share an architecture. To our knowledge, MiniCPM-o 4.5 is the only other publicly available full-duplex speech model that supports sliding-window. It differs from Moshi in its backbone (a Qwen3-8B-based LLM). We added experiments on this model in the new Appendix E.

  • The collapse pattern of Moshi and PersonaPlex is reproduced. As in Moshi and PersonaPlex, collapse onset tracks window length. With a FIFO sliding window, gibberish occurred in 61/63 runs, always after the first eviction. Its onset was delayed as the window grew (Fig. 21).
  • Keeping only the first entry is enough. IAP, which keeps only the first entry, gave failure rates and response quality comparable to MiniCPM-o 4.5 (Figs. 22–23). It always keeps the initial prompt (the first 147 KV entries) in the cache and evicts only the entries after it. Thus, keeping a single entry, instead of all 147, is enough to mitigate collapse.
  • DRP reduces prolonged silence. Adding DRP to IAP reduced prolonged silence from 40/40 to 14/40 replays and raised mean Coherence from 1.26 to 3.29 and mean Context from 1.21 to 2.15 (Figs. 22–23).

Together, these results suggest that both our diagnosis and our fixes extend beyond the Moshi family.

AuthorsOpenReview ↗

General Response 2

GR2. Analysis of growing-context models

Most full-duplex speech models have focused on real-time interaction rather than long conversations. So many models use a growing-context approach, which keeps all previous tokens in the KV cache. However, this approach is known to suffer from two problems in long conversations: memory usage grows with conversation length, and generation quality can degrade once the context exceeds the lengths seen during training [1–3]. Therefore, a sliding-window approach is better suited to long conversations, as it keeps only the most recent tokens and thus bounds both memory usage and the distance between tokens. Prior work on long streaming inference has accordingly used sliding windows [2, 4, 5].

Growing-context models also degrade. In our experiments, BayLing-Duplex and Lychee-FD collapsed before reaching their context limits, and Raon-SpeechChat ran out of context after about 5 min (Fig. 7). One likely factor is short training sequences. BayLing-Duplex’s public source dataset, InstructS2S-200K, contains an average of 39.29 seconds per dialogue, and Raon-SpeechChat uses at most 4,096 tokens (about 2 min) in full-duplex training. As a result, these models see only short distances between tokens during training. According to [1], RoPE attention depends on the relative distance between tokens, and models can fail once this distance exceeds the range seen during training.

Relative distance between tokens affects BayLing-Duplex collapse. Motivated by [1], we changed the spacing between position indices in BayLing-Duplex, which changes the relative distance between the newest and oldest tokens in the context (Appendix F, Table 20). Halving the spacing (0, 0.5, 1.0, ...) brought tokens closer together and delayed prolonged silence from 7.8 to 11.4 min. This shows that relative distance matters for BayLing-Duplex collapse. Following the reviewers’ suggestions, we added this analysis to Appendix F.

A sliding window with IAP and DRP on BayLing-Duplex reduces collapse and improves response quality. We evaluated BayLing-Duplex with a 120-second sliding window, IAP, and DRP on all 40 FD-LHB-LLM sessions.

Inference settingDurationRepetitionProlonged silenceGibberishCoherenceContext
BayLing-Duplex (Growing context)17 min 28 s1/4040/4040/401.181.46
BayLing-Duplex (Sliding window) + IAP + DRP20 min0/400/4013/402.263.01

Without any retraining, the sliding window with IAP and DRP let all 40 sessions run the full 20 minutes, beyond BayLing-Duplex’s context limit (17 min 28 s). Prolonged silence, which affected every growing-context session, disappeared entirely (40/40 → 0/40), as did repetition (1/40 → 0/40), and gibberish fell by about two-thirds (40/40 → 13/40). Response quality roughly doubled: over the first 17 minutes, which both settings cover, mean Coherence rose from 1.18 to 2.26 and Context from 1.46 to 3.01 (Appendix F, Figs. 24–25). These results suggest that our training-free fixes can benefit even models trained with a growing context.

[1] Chen, Shouyuan, et al. "Extending context window of large language models via positional interpolation." arXiv preprint arXiv:2306.15595 (2023).

[2] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." International Conference on Learning Representations. Vol. 2024. 2024.

[3] Xiao, Chaojun, et al. "Infllm: Training-free long-context extrapolation for llms with an efficient context memory." Advances in neural information processing systems 37 (2024): 119638-119661.

[4] Ning, Zhenyu, et al. "Inf-mllm: Efficient streaming inference of multimodal large language models on a single gpu." arXiv preprint arXiv:2409.09086 (2024).

[5] Di, Shangzhe, et al. "Streaming video question-answering with in-context video kv-cache retrieval." International Conference on Learning Representations. Vol. 2025. 2025.

GR3. Summary of revisions

  • Added MiniCPM-o 4.5 experiments with IAP and DRP (Appendix E; 2re1, 4t14, s4qy).
  • Added BayLing-Duplex analysis and sliding-window results with IAP and DRP (Appendix F; 2re1, 4t14).
  • Added Full-Duplex-Bench evaluation after cache eviction (Appendix G; 4t14).
  • Added separate diagnostic results for Moshi and PersonaPlex (Appendix B.2; s4qy).
  • Extended the DRP ablation to 40 replays and added a hyperparameter sensitivity analysis (Appendices D.5-D.7).
  • Updated the limitations to reflect the new experiments (Appendix H.2).
AuthorsOpenReview ↗

Summary of Discussion for the AC

We thank the AC and reviewers for their constructive feedback.

Paper in brief

We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which evaluates five full-duplex speech models using replays of up to 60 minutes and adaptive sessions of up to 20 minutes. Within these horizons, every baseline either exhibits at least one of the three failure modes illustrated in Fig. 1—repetition, prolonged silence, and gibberish—or exhausts its context. For sliding-window models, our experiments link silence and gibberish to eviction of the first KV entry and show that recent outputs reinforce repetition.

These findings motivate two training-free interventions: Initialization-Anchor Pinning (IAP) retains the first entry, and Duplex Repetition Penalty (DRP) discourages repeated words while guarding against silence. On 40 replays per model, IAP+DRP reduces gibberish from 40/40 to 0/40 in both Moshi and PersonaPlex; repetition falls from 26/40 to 0/40 and 36/40 to 1/40, respectively. Prolonged silence also falls from 40/40 to 3/40 and 34/40 to 19/40. The method runs in real time on PersonaPlex (RTF 0.75).

Strengths and concerns addressed

Reviewers recognized the importance of this under-studied problem, the diagnostic experiments, and the simple, efficient interventions. All three rated the presentation clear and easy to follow.

  • Generality beyond Moshi (2re1, 4t14, s4qy): To test whether our findings extend beyond the Moshi family, we added MiniCPM-o 4.5, which uses a different backbone (Qwen3-8B). Gibberish appeared in 61/63 FIFO runs, always after eviction of the first entry. Retaining that entry alone performed comparably to retaining its full 147-entry initial prompt; adding DRP reduced prolonged silence from 40/40 to 14/40 and raised mean Coherence from 1.26 to 3.29 (GR1; App. E).

  • Growing-context models (2re1, 4t14): To extend our analysis beyond models that natively use a sliding window, we examined why the growing-context BayLing-Duplex collapses and tested whether converting it to sliding-window inference with IAP+DRP could help. Halving positional spacing delayed prolonged silence from 7.8 to 11.4 minutes, implicating relative token distance. A 120-second sliding window with IAP+DRP let all 40 sessions reach 20 minutes, beyond the original 17.5-minute context limit. Prolonged silence fell from 40/40 to 0/40 and gibberish from 40/40 to 13/40; Coherence and Context roughly doubled over the period covered by both settings (GR2; App. F).

  • Full-duplex behavior (4t14): To assess whether the interventions disrupt full-duplex interaction, we evaluated turn-taking, interruptions, pauses, and backchannels on Full-Duplex-Bench v1. At session start, DRP did not substantially change smooth turn-taking, user-interruption takeover rates, or backchannel frequency. It increased Moshi's backchannel takeover rate; our follow-up found some genuine takeovers, while most newly flagged responses were short acknowledgments counted as takeovers by the scorer. After 20 minutes of silent-input history, IAP+DRP preserved turn-taking while both baselines largely stopped taking turns (App. G).

  • Component roles (4t14): We clarified that IAP and DRP play complementary roles. In Moshi, DRP alone leaves gibberish at 40/40, while IAP removes it (App. D.4). In a matched 20-minute PersonaPlex ablation, repetition fell from 34/40 with IAP alone to 0/40 with IAP+DRP (App. D.5).

  • Comparison with training (4t14): Following 4t14's suggestion to position our training-free approach against lightweight fine-tuning, we fine-tuned PersonaPlex with LoRA and evaluated it on the first 20 minutes of the same 40 replays. No LoRA setting resolved all three failure types together, whereas IAP+DRP, without any training, reached 0/40 repetition, 5/40 prolonged silence, and 0/40 gibberish (response to 4t14).

  • Evaluation clarifications (2re1, 4t14, s4qy): We added per-model diagnostics to the manuscript (App. B.2). In our responses, we clarified the text-based metrics and pointed to their audio validation (Apps. A.2–A.4), explained that the replay track preserves natural overlap whereas the adaptive track is turn-based, and identified the pinned first entry as a BOS initialization frame rather than a prompt.

Discussion status and limits

Reviewer 4t14 said our response “addressed most of my concerns” and increased their rating to Weak acceptance; reviewer s4qy maintained a positive assessment. Reviewer 2re1's two principal concerns, generality and growing-context analysis, are addressed in GR1 and GR2. Remaining limitations include DRP's early backchannel trade-off and late gibberish in 2/40 Moshi adaptive sessions. Our claim is substantial mitigation within the tested horizons, rather than failure-free long conversations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.