- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the speech-text token rate difference in current SpeechLLMs. For the same spoken content, the number of speech tokens is often times more than the number of text tokens, which creates the "gap". This gap also depends on the frame rate of different audio codec tokens. This work proposed to compress reasoning traces into these gaps to utilize them. They evaluate their model on spoken math problems, which shows an accuracy increase especially in higher audio frame rate case. They also conducted several ablation experiments to show the effect of each design choice.
Critical strengths
- It is the first paper to bring focus to the "gap" between text and speech token lengths in SLMs and their relationship to different audio codec frame rates.
- The application of text compression into these text gaps is novel and their evaluation show that the method is working.
Critical weaknesses
-
Over-claim the contribution using the speech-text gaps for thinking in the introduction.
I acknowledge the novelty of bringing dedicated focus on the discussion of the gap with respect to different audio frame rates. However, many prior works in the field of reasoning + SpeechLLMs already mention this phenomenon and they achieve thinking while listening or thinking while speaking based on this gap.
-
The scope of this method seems to be limited to parallel speech text generation (E.g. Moshi style, multiple streams of token are stacked. Model predicts multiple tokens from each stream each backbone step).
The other line of work interleaved different streams (speech/text) into chunks. Hence, there are no so-called "idle" positions caused by the speech and text tokens’ length mismatch.
-
The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.
According to Table 1 and the overall setup of the paper, we see that the higher the audio codec frame rate, the larger the gap between speech and text, thus there is more idle spaces to insert thinking. And the result shows that with a higher audio codec frame rate, the accuracy improves more. However, in practice, if I understand correctly, the hardware calculation is bounded. This indicates that with the same model on the same hardware, there is a limit on the decoding speed. With a higher audio codec frame rate, this means that in the same amount of time, the model has to decode more tokens. Wouldn't this decrease the capacity for decoding assistant text and compressed thinking tokens?
-
The manuscript is hard to follow.
-
For Fig1, the left-hand side model diagram is not clear. As a reader, it is hard for me to connect the model diagram and the token alignments on the right. The Figure description also doesn't explain the relation. For the (a) (b) (c), I would suggest drawing a timeline so that the reader can better understand the alignment between speech and text tokens.
-
What is "group size" in Figure1? Perhaps I miss something, but I cannot find a formal definition of group size in the manuscript.
-
Not sure if I'm understanding correctly, Standard Parallel is aligned on the entire spoken turn of the assistant. Segment alignment is aligned on each utterance. I infer this from the figure 1 example. I would suggest this being clear stated in the method section.
-
Questions for authors
- How does your backbone model decodes speech and text tokens? Do you decode in parallel (multiple heads attached to the backbone?) What is the ratio? One speech one text? I guess this has something to do with the group size? I strongly suggest you make it clear both in Figure1 and the manuscript.
- Please provide standard deviation or confidence intervals for the tables. I would like to know whether the accuracies and latencies changes are meaningful.
- Line 197, for "(D2) shared conditioning", how serious is this problem? Does the conditional speech head learn the relation of padding text tokens? I guess padding tokens even though they don’t carry semantic information, they indicate timing information?
- Different audio frame rates give different gaps, how does the model know how much reasoning to decode given different frame rates?
- What is the reason for "Alignment" method to have different accuracies compared to "baseline" method. Both setup doesn't involve reasoning right?
5 comments
Q4. For (D2) shared conditioning.
- Severity of the problem: We conducted an experiment on this issue in Appendix C. As shown in Fig. 4, accuracy consistently decreases without gating (w/o invisible) compared with gating (w/ invisible).
- Your assessment is correct that idle tokens carry timing information. In pre-training, idle tokens appear only at the end and are highly co-occurrent with the audio entering its concluding portion. The speech head can therefore use “the current position is an idle token” as a proxy signal for elapsed playback time or segment progress. Accordingly, the gate is designed to be minimally invasive: it only zeros the text embedding of the current slot provided to the speech head while fully preserving h_{n−1}. Timing information remains available through the backbone hidden-state pathway. This point was omitted from the analysis in the original manuscript, and we thank the reviewer for pointing it out.
We thank you again for your thorough review and for recognizing the novelty of our compressed reasoning approach. We hope these clarify your concerns, and further discussion is welcome.
Thanks for all the efforts of the authors for providing additional informations and the improvement of the paper's readability.
After reading the authors' response, I have a clearer understanding of the model architecture.
However, I still have several questions regarding the authors' response.
Q1. About novelty.
I agree on the novelty of reasoning compression method for step-wise reasoning and train the model to adaptively control reasoning length proportionally. However, I still have questions for the author's response.
-
Even though this paper work on thinking-while-speaking, papers that work on thinking-while-listening often time share the similar underlying architecture techniques. The difference is that whether the model utilize the "pad" during either the user is speaking or the assistant is speaking. The method of utilizing the "pad" tokens is roughly the same. It would be better if the authors also mentioned these in the paper if it makes sense.
I listed out three related works that utilize "pad" in the assistant text stream for reasoning.
-
Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, and Mike Seltzer. "Can Speech LLMs Think while Listening?" arXiv preprint arXiv:2510.07497, 2025. https://arxiv.org/abs/2510.07497
I saw the authors already cited this as a related work in their paper. In this paper, the authors utilize the "pad" tokens while the user is speaking for doing streaming ASR of the user and the thinking.
-
Donghang Wu, Haoyang Zhang, Chen Chen, Tianyu Zhang, Fei Tian, Xuerui Yang, Gang Yu, Hexin Liu, Nana Hou, Yuchen Hu, and Eng Siong Chng. "Chronological Thinking in Full-Duplex Spoken Dialogue Language Models." arXiv preprint arXiv:2510.05150, 2025. https://arxiv.org/abs/2510.05150
In this paper, the authors utilize the "pad" tokens on the assistant text channel for reasoning while the user is speaking.
-
Donghang Wu, Tianyu Zhang, Yuxin Li, Hexin Liu, Chen Chen, Eng Siong Chng, and Yoshua Bengio. "The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning." arXiv preprint arXiv:2603.17837, 2026. https://arxiv.org/abs/2603.17837
In this paper, the authors do latent reasoning in the assistant text channel while the user is speaking. I think this is also highly related to the current paper. In general, both paper are trying to compress the reasoning.
-
-
The author mention that for prior works on thinking-while-speaking, the cost stays hidden only if the hardware generates faster than playback
I'm not fully understanding the difference from this work. Isn't it true that for all systems to be streaming/realtime the hardware must generate faster than the playback?
In conclusion, I think that the current narrative is somehow limited. There are different existing ways of leveraging the gap between audio and text. Using "pad" token in assistant text is one category. Adding additional streams like DuplexSLA is also another way. (They flatten into one token sequence and make sure the real-time constraints are met. The reason they can do this is because the gap between audio and text). I hope the authors can consider citing and mentioning these related prior works and make the novelty of this work clearer and stronger.
Q2. The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.
My concern is that without the concept of model decoding rate limit, Table1 gives the reader the takeaway that using a higher audio frame rate provides more idle slots for text tokens. Therefore, we have more spaces for reasoning chains, leading to more improvement in accuracy. This sounds weird to me. Under a limit of decoding rate, doesn't this mean that if we choose a lower audio frame rate, we can generate less audio tokens and we can use the rest of the decoding capacity for generating more text tokens? After reading the author responses, I think the main constraint in this paper is that the tokens prediction scheme is G:1 for audio : text tokens? This means that this model architecture doesn't allow a higher rate for text tokens than the audio? Or is there any significant advantage for choosing a higher audio frame rate codec? I hope the author can put more discussion on this in the paper.
Q4. For (D2) shared conditioning.
For the experiments you mentioned in Appendix C, are you training one model and inference with difference gating setup or you are training independent models?
Response to Reviewer rhVf
We appreciate your thorough review and valuable feedback. Several of your questions are covered in the General Response:
Responses to the remaining questions follow below.
Q1. About novelty.
We think a clarification of the novelty is warranted.
We will add the above distinction to the Introduction.
Q2. The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.
We agree that the manuscript does not consider decoding rate limits. When comparing across frame rates, lower-frame-rate models indeed save resources: they require fewer LLM forward passes (Table 3) and have shorter end-to-end generation times (Table 4). Therefore, if the number of decoding steps or the decoding time is limited (e.g., max generation length or max generation time), a high-frame-rate model may be truncated before generating the complete content, reducing accuracy. However, this does not imply that, when the model can complete inference normally, its ability to decode assistant text and compressed thinking tokens is diminished.
Additionally, we have supplemented the discussion on speech decoding in G3. Your concern about system-level decoding limits is valid, and we will incorporate this point in the Method section. However, it is important to note that the decoding rate limitation is inherent to high-frame-rate speech LLMs under the standard paradigm, rather than being introduced by our method.
Q3. Figure Revisions and Concept Clarifications.
Thank you for the constructive suggestions. We will improve Fig. 1 by adding token-category annotations to the model architecture diagram (left) and adding a timeline (right).
Group size is a hyperparameter through which Fun-Audio-Chat implements different speech output frame rates: each forward pass of the backbone model generates one group of speech tokens, whose number is the group size. Setting this hyperparameter determines the model's speech output frame rate. We will add this explanation to the figure caption.
At each inference step, the backbone model generates one hidden state, from which the text head generates one text token and the speech head autoregressively generates g speech tokens (forming one group, where g is the group size). The two output heads decode in parallel at a ratio of 1 : g. We will explain this process more clearly in “Baseline parallel generation” in Section 3.1, Background.
Your interpretation of segment alignment is accurate, and we will add clarification to the relevant section.
Regarding "How does the model know how much reasoning to decode given different frame rates?", the audio frame rate f_a is a model hyperparameter, with each frame rate corresponding to an independent SFT checkpoint rather than a rate-agnostic model. Furthermore, the mechanism for reasoning termination is in Section G6.