- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates the long-horizon generation problem in full-duplex speech LLMs. The authors developed the Full-Duplex Long-Horizon Benchmark (FD-LHB) to evaluate five recent full-duplex models and found that these models tend to collapse during long-horizon inference, such as 30- or 60-minute sessions. The authors then focus on the repetition, silence, and gibberish problems in sliding-window full-duplex models. They propose Initialization-Anchor Pinning (IAP) to address gibberish and silence, and Duplex Repetition Penalty (DRP) to mitigate repetition. The results show that both proposed techniques are highly effective in addressing their respective target problems.
Critical strengths
- Strong Novelty: This is one of the most interesting papers I have read recently. Most existing full-duplex works focus on model training, developing new model architectures or collecting more data to improve model performance. However, to the best of my knowledge, no existing work has successfully addressed the long-horizon collapse problem in full-duplex models. This paper approaches the problem from a different angle: inference-time intervention. Full-duplex models are known to be highly sensitive to inference parameters, so I personally believe that inference-time manipulation (e.g., through the inference harness) is a promising research direction for this field.
- Strong Results: The overall results demonstrate that the proposed methods are highly effective. The latency and RTF analyses are also crucial for demonstrating their practical applicability.
- Good Presentation and Visualization: The paper includes numerous figures and charts that clearly illustrate both the problems and the effectiveness of the proposed methods.
Critical weaknesses
- Somewhat Limited Backbones: The authors mainly evaluate their proposed methods on Moshi and PersonaPlex, which belong to the same model family (PersonaPlex is a fine-tuned version of Moshi). I understand that there may not be many sliding-window models available, but I wonder whether there are other similar models on which the proposed methods could be evaluated.
- Combined Results: Some figures and tables aggregate the results from Moshi and PersonaPlex. This makes the results less informative, as it is possible that only one model exhibits the expected improvements while the other does not, and averaging their results could obscure such differences.
Questions for authors
- I suspect that the first entry in PersonaPlex might correspond to a system prompt or voice prompt token rather than a conversational token. What exactly is preserved in the first KV entry for PersonaPlex?
- For PersonaPlex, the silence rate only decreases from 85% to 47.5%, which is a less substantial improvement compared to Moshi (from 100% to 7.5%). Have the authors analyzed why the improvement is less pronounced for PersonaPlex?
2 comments
Thank the authors for their rebuttal. I maintain my positive assessment after reviewing the rebuttal.
We sincerely thank the reviewer for the very positive assessment, and for recognizing the novelty of our inference-time approach, the strength of the results, and the clarity of the presentation.
W1. Limited backbones
Thank you for the suggestion. We extended our evaluation to MiniCPM-o 4.5, which uses a different architecture from Moshi and PersonaPlex, to further examine the generalizability of our findings. Please see GR1 for details.
W2. Per-model results for Moshi and PersonaPlex
Thank you for the suggestion. We now show separate results for Moshi and PersonaPlex in Appendix B.2 (Figs. 14–17, Tables 9–10). These results show that the main findings hold for both models.
Q1. Contents of PersonaPlex's first KV entry
The first KV entry in PersonaPlex comes from a BOS initialization frame, not a system prompt, voice prompt, or conversation. This frame contains one text BOS token and 16 audio BOS tokens. The voice and text prompts come after this frame. Moshi also starts with the same type of BOS frame. Thus, IAP keeps the initialization state, not the prompt information, in both models.
Q2. Smaller silence reduction in PersonaPlex
Most remaining silences in PersonaPlex are temporary. With IAP+DRP, 19/40 runs still had a two-minute silence, but 18 of them resumed speaking. We have not identified a clear cause, but one possible factor is training data. In its final stage, Moshi was trained only on LLM-written dialogues rendered with TTS, in which it mostly responds to the user and the user rarely pauses mid-utterance. PersonaPlex’s final training also includes real phone conversations (Fisher), which contain natural pauses and listening, so it may stay silent more often.