acceptodds
Under review as a conference paper at ICLR 2027

Thinking in the Gaps: Reusing the Text–Speech Frame-Rate Gap for Chain-of-Thought Reasoning

Abstract

Spoken language models (SLMs) enable natural human-computer interaction, but most decode text and speech at different frame rates on a shared autoregressive grid, leaving many text positions idle while speech generation continues. We introduce Thinking in the Gaps, which reuses these idle positions for stepwise reasoning in parallel with speech generation. Because full reasoning traces often exceed the available gap capacity, a context-conditioned compressor constructs length-constrained reasoning targets that fit within these gaps. Across five spoken mathematical question-answering benchmarks, this method improves average accuracy over the matched baseline at every tested frame rate, with the largest gain of 14.05 percentage points at 25 Hz. Crucially, these gains come at virtually no additional inference cost and without any interruption to speech generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spoken Language Models, Chain-of-Thought, Efficient Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rhVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the speech-text token rate difference in current SpeechLLMs. For the same spoken content, the number of speech tokens is often times more than the number of text tokens, which creates the "gap". This gap also depends on the frame rate of different audio codec tokens. This work proposed to compress reasoning traces into these gaps to utilize them. They evaluate their model on spoken math problems, which shows an accuracy increase especially in higher audio frame rate case. They also conducted several ablation experiments to show the effect of each design choice.

Critical strengths
  1. It is the first paper to bring focus to the "gap" between text and speech token lengths in SLMs and their relationship to different audio codec frame rates.
  2. The application of text compression into these text gaps is novel and their evaluation show that the method is working.
Critical weaknesses
  1. Over-claim the contribution using the speech-text gaps for thinking in the introduction.

    I acknowledge the novelty of bringing dedicated focus on the discussion of the gap with respect to different audio frame rates. However, many prior works in the field of reasoning + SpeechLLMs already mention this phenomenon and they achieve thinking while listening or thinking while speaking based on this gap.

  2. The scope of this method seems to be limited to parallel speech text generation (E.g. Moshi style, multiple streams of token are stacked. Model predicts multiple tokens from each stream each backbone step).

    The other line of work interleaved different streams (speech/text) into chunks. Hence, there are no so-called "idle" positions caused by the speech and text tokens’ length mismatch.

  3. The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.

    According to Table 1 and the overall setup of the paper, we see that the higher the audio codec frame rate, the larger the gap between speech and text, thus there is more idle spaces to insert thinking. And the result shows that with a higher audio codec frame rate, the accuracy improves more. However, in practice, if I understand correctly, the hardware calculation is bounded. This indicates that with the same model on the same hardware, there is a limit on the decoding speed. With a higher audio codec frame rate, this means that in the same amount of time, the model has to decode more tokens. Wouldn't this decrease the capacity for decoding assistant text and compressed thinking tokens?

  4. The manuscript is hard to follow.

    • For Fig1, the left-hand side model diagram is not clear. As a reader, it is hard for me to connect the model diagram and the token alignments on the right. The Figure description also doesn't explain the relation. For the (a) (b) (c), I would suggest drawing a timeline so that the reader can better understand the alignment between speech and text tokens.

    • What is "group size" in Figure1? Perhaps I miss something, but I cannot find a formal definition of group size in the manuscript.

    • Not sure if I'm understanding correctly, Standard Parallel is aligned on the entire spoken turn of the assistant. Segment alignment is aligned on each utterance. I infer this from the figure 1 example. I would suggest this being clear stated in the method section.

Questions for authors
  1. How does your backbone model decodes speech and text tokens? Do you decode in parallel (multiple heads attached to the backbone?) What is the ratio? One speech one text? I guess this has something to do with the group size? I strongly suggest you make it clear both in Figure1 and the manuscript.
  2. Please provide standard deviation or confidence intervals for the tables. I would like to know whether the accuracies and latencies changes are meaningful.
  3. Line 197, for "(D2) shared conditioning", how serious is this problem? Does the conditional speech head learn the relation of padding text tokens? I guess padding tokens even though they don’t carry semantic information, they indicate timing information?
  4. Different audio frame rates give different gaps, how does the model know how much reasoning to decode given different frame rates?
  5. What is the reason for "Alignment" method to have different accuracies compared to "baseline" method. Both setup doesn't involve reasoning right?
5 comments
AuthorsOpenReview ↗

Response to Reviewer rhVf

We appreciate your thorough review and valuable feedback. Several of your questions are covered in the General Response:

  • standard errors and confidence intervals for accuracy (G1, Tables R1–R2) and latency (G2, Tables R3–R5),
  • the accuracy difference between Alignment and Baseline (G1),
  • real-time factor under the hardware decoding limit (G3, Table R7),
  • and how the model determines when to stop reasoning (G6).

Responses to the remaining questions follow below.


Q1. About novelty.

We think a clarification of the novelty is warranted.

  1. The distinction from previous think-while-speaking work
  • Reasoning during speech output has been explored before. These methods exploit a wall-clock gap: reasoning tokens are additional sequence positions, and their cost stays hidden only if the hardware generates faster than playback. However, ThinkGap exploits a structural gap inside the parallel generation grid: the text slots that the backbone already computes at every step and that the Baseline fills with padding. For a given spoken response, reasoning adds no sequence positions, and the reasoning capacity is fixed by the frame rate rather than by hardware headroom.
  • Furthermore, we design a reasoning compression method for step-wise reasoning and train the model to adaptively control reasoning length proportionally. These two contributions are absent from existing thinking-while-speaking work.
  1. The scope of this method
  • Our method is indeed designed for parallel-architecture SLMs. However, as noted in the original Introduction, this represents a mainstream architecture upon which numerous SLM works are built. More importantly, these architectures commonly waste substantial resources on idle tokens in the text stream. By reducing this waste and more fully utilizing model capabilities, our method offers significant value.

We will add the above distinction to the Introduction.


Q2. The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.

We agree that the manuscript does not consider decoding rate limits. When comparing across frame rates, lower-frame-rate models indeed save resources: they require fewer LLM forward passes (Table 3) and have shorter end-to-end generation times (Table 4). Therefore, if the number of decoding steps or the decoding time is limited (e.g., max generation length or max generation time), a high-frame-rate model may be truncated before generating the complete content, reducing accuracy. However, this does not imply that, when the model can complete inference normally, its ability to decode assistant text and compressed thinking tokens is diminished.

Additionally, we have supplemented the discussion on speech decoding in G3. Your concern about system-level decoding limits is valid, and we will incorporate this point in the Method section. However, it is important to note that the decoding rate limitation is inherent to high-frame-rate speech LLMs under the standard paradigm, rather than being introduced by our method.


Q3. Figure Revisions and Concept Clarifications.

  1. Thank you for the constructive suggestions. We will improve Fig. 1 by adding token-category annotations to the model architecture diagram (left) and adding a timeline (right).

  2. Group size is a hyperparameter through which Fun-Audio-Chat implements different speech output frame rates: each forward pass of the backbone model generates one group of speech tokens, whose number is the group size. Setting this hyperparameter determines the model's speech output frame rate. We will add this explanation to the figure caption.

  3. At each inference step, the backbone model generates one hidden state, from which the text head generates one text token and the speech head autoregressively generates g speech tokens (forming one group, where g is the group size). The two output heads decode in parallel at a ratio of 1 : g. We will explain this process more clearly in “Baseline parallel generation” in Section 3.1, Background.

  4. Your interpretation of segment alignment is accurate, and we will add clarification to the relevant section.

  5. Regarding "How does the model know how much reasoning to decode given different frame rates?", the audio frame rate f_a is a model hyperparameter, with each frame rate corresponding to an independent SFT checkpoint rather than a rate-agnostic model. Furthermore, the mechanism for reasoning termination is in Section G6.


Q4. For (D2) shared conditioning.

  1. Severity of the problem: We conducted an experiment on this issue in Appendix C. As shown in Fig. 4, accuracy consistently decreases without gating (w/o invisible) compared with gating (w/ invisible).
  2. Your assessment is correct that idle tokens carry timing information. In pre-training, idle tokens appear only at the end and are highly co-occurrent with the audio entering its concluding portion. The speech head can therefore use “the current position is an idle token” as a proxy signal for elapsed playback time or segment progress. Accordingly, the gate is designed to be minimally invasive: it only zeros the text embedding of the current slot provided to the speech head while fully preserving h_{n−1}. Timing information remains available through the backbone hidden-state pathway. This point was omitted from the analysis in the original manuscript, and we thank the reviewer for pointing it out.

We thank you again for your thorough review and for recognizing the novelty of our compressed reasoning approach. We hope these clarify your concerns, and further discussion is welcome.

Thanks for all the efforts of the authors for providing additional informations and the improvement of the paper's readability.

After reading the authors' response, I have a clearer understanding of the model architecture.

However, I still have several questions regarding the authors' response.

Q1. About novelty.

I agree on the novelty of reasoning compression method for step-wise reasoning and train the model to adaptively control reasoning length proportionally. However, I still have questions for the author's response.

  • Even though this paper work on thinking-while-speaking, papers that work on thinking-while-listening often time share the similar underlying architecture techniques. The difference is that whether the model utilize the "pad" during either the user is speaking or the assistant is speaking. The method of utilizing the "pad" tokens is roughly the same. It would be better if the authors also mentioned these in the paper if it makes sense.

    I listed out three related works that utilize "pad" in the assistant text stream for reasoning.

    • Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, and Mike Seltzer. "Can Speech LLMs Think while Listening?" arXiv preprint arXiv:2510.07497, 2025. https://arxiv.org/abs/2510.07497

      I saw the authors already cited this as a related work in their paper. In this paper, the authors utilize the "pad" tokens while the user is speaking for doing streaming ASR of the user and the thinking.

    • Donghang Wu, Haoyang Zhang, Chen Chen, Tianyu Zhang, Fei Tian, Xuerui Yang, Gang Yu, Hexin Liu, Nana Hou, Yuchen Hu, and Eng Siong Chng. "Chronological Thinking in Full-Duplex Spoken Dialogue Language Models." arXiv preprint arXiv:2510.05150, 2025. https://arxiv.org/abs/2510.05150

      In this paper, the authors utilize the "pad" tokens on the assistant text channel for reasoning while the user is speaking.

    • Donghang Wu, Tianyu Zhang, Yuxin Li, Hexin Liu, Chen Chen, Eng Siong Chng, and Yoshua Bengio. "The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning." arXiv preprint arXiv:2603.17837, 2026. https://arxiv.org/abs/2603.17837

      In this paper, the authors do latent reasoning in the assistant text channel while the user is speaking. I think this is also highly related to the current paper. In general, both paper are trying to compress the reasoning.

  • The author mention that for prior works on thinking-while-speaking, the cost stays hidden only if the hardware generates faster than playback

    I'm not fully understanding the difference from this work. Isn't it true that for all systems to be streaming/realtime the hardware must generate faster than the playback?

In conclusion, I think that the current narrative is somehow limited. There are different existing ways of leveraging the gap between audio and text. Using "pad" token in assistant text is one category. Adding additional streams like DuplexSLA is also another way. (They flatten into one token sequence and make sure the real-time constraints are met. The reason they can do this is because the gap between audio and text). I hope the authors can consider citing and mentioning these related prior works and make the novelty of this work clearer and stronger.

Q2. The concept of model decoding rate limit is not considered together with the "thinking gap" concept in the paper.

My concern is that without the concept of model decoding rate limit, Table1 gives the reader the takeaway that using a higher audio frame rate provides more idle slots for text tokens. Therefore, we have more spaces for reasoning chains, leading to more improvement in accuracy. This sounds weird to me. Under a limit of decoding rate, doesn't this mean that if we choose a lower audio frame rate, we can generate less audio tokens and we can use the rest of the decoding capacity for generating more text tokens? After reading the author responses, I think the main constraint in this paper is that the tokens prediction scheme is G:1 for audio : text tokens? This means that this model architecture doesn't allow a higher rate for text tokens than the audio? Or is there any significant advantage for choosing a higher audio frame rate codec? I hope the author can put more discussion on this in the paper.

Q4. For (D2) shared conditioning.

For the experiments you mentioned in Appendix C, are you training one model and inference with difference gating setup or you are training independent models?

Reviewer_zXQfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes THINKGAP for spoken language models that generate text and speech in parallel. It uses idle text positions for silent reasoning while speech generation continues. An offline compressor selects reasoning tokens to fit these positions, and the model learns to generate compressed reasoning directly.

The method is evaluated on five spoken math QA benchmarks and five frame rates. For the 1.7B model, the average gain is 6.03 percentage points across rates, with the largest gain of 14.05 points at 25 Hz. Average latency stays close to baseline. Results on larger models are mixed.

Critical strengths
  1. The method makes useful use of existing decoding positions. The compressor is only used to prepare training data, so it adds no compressor cost during inference.

  2. The improvement in the main 1.7B setting is substantial. At 25 Hz, accuracy increases from 54.68 to 68.73, while alignment alone gives 54.78 (Table 1). Compared with full reasoning, THINKGAP loses only 0.60 points and uses about 20.7% fewer decoding steps (Table 3).

Critical weaknesses
  1. The efficiency claim needs qualification. The reported average cost is close to baseline, but the global budget correction and inference stopping rule are not fully explained. First speech-token latency also does not establish uninterrupted audio playback. These gaps limit the strongest efficiency claims. The authors should clarify the procedure and provide timing evidence for playable audio.

  2. The benefit depends on the setting. At 8B/5 Hz, THINKGAP gives 75.71 versus 75.76 for the adapted baseline (Table 2), although the 8B model improves by 3.37 points at 25 Hz. The released 5 Hz model scores 83.99; its training differs, but the adaptation drop needs explanation. Separately, the 1.7B model's WER at 5 Hz rises from 0.100 to 0.188. The role of number formatting is unclear. These results limit claims about scaling and speech fidelity.

Questions for authors
  1. How are the global budget correction and reasoning stopping rule implemented? What happens if reasoning is still unfinished when the speech segment ends?
  2. Does the latency measurement include waveform decoding and buffering? Can you report first playable audio latency and pauses between audio packets?
  3. Why does adaptation reduce the 8B/5 Hz baseline accuracy so much? Separately, does the 1.7B WER increase remain after normalizing number formats?
  4. Can you provide uncertainty estimates for the smaller gains? At 25 Hz, random token selection already gives 68.08 versus 68.73, so the extra benefit of the proposed compressor is less clear.
2 comments
AuthorsOpenReview ↗

Response to Reviewer zXQf

We appreciate your thorough review and valuable feedback. The major concerns you highlighted are covered in the General Response:

  • global budget correction (G5),
  • the reasoning stopping rule and the case where speech ends before reasoning (G6),
  • first-playable-audio latency (G2, Table R3),
  • pauses in the generated audio and real-time factor (G3, Tables R6–R7),
  • uncertainty estimates (G1),
  • and WER after number normalization (G4).

Responses to the remaining questions follow below, and we hope these clarify all your concerns.


Q1. Accuracy degradation of the 8B/5 Hz baseline.

Two factors may account for this decline: (i) SFT-induced forgetting of capabilities acquired through multi-task DPO training (in the Fun-Audio-Chat Technical Report); and (ii) discrepancies in data scale and training recipe compared to the publicly released model. We agree that clarifying the underlying causes of the 8B/5 Hz performance drop is necessary and will incorporate this explanation into Section 4.3 of the manuscript.


Q2. Limited improvement over random selection at 25 Hz

We agree that the improvement over random selection at 25 Hz is limited. At this frame rate, the large speech–text gap leaves a generous reasoning budget: most CoT tokens are retained, and nearly half of the segments require no compression. Random deletion and context-conditioned selection therefore have similar effects. At lower frame rates, tighter budgets make token selection more consequential, consistent with the larger point-estimate gains from context-conditioned selection in Table 3. Table R9 summarizes these compression statistics.

Table R9: CoT compression statistics for the ThinkGap training data across frame rates.

Frame rateMean retention ratio per CoT segmentSegments requiring no compression
25.0 Hz63.73%48.07%
12.5 Hz38.65%21.64%
8.33 Hz24.19%10.18%
6.25 Hz14.96%4.86%
5.00 Hz10.06%2.34%
AuthorsOpenReview ↗

Response to Reviewer zXQf

As the author–reviewer discussion period is about to end, we would like to kindly follow up. We would appreciate any feedback on whether our responses have resolved your concerns, and we are glad to clarify anything further before the deadline.

Thank you again for your detailed review.

Reviewer_Zjq5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes THINKGAP, a method for using otherwise idle text positions in parallel speech–text generation for intermediate reasoning. The method distributes these positions between spoken segments and fills them with compressed reasoning for the next segment. Training targets are constructed using context-conditioned extractive compression, while a gating mechanism prevents non-spoken tokens from directly conditioning the speech head at the current position. No separate compressor is required at inference time.

Experiments cover five spoken mathematical QA benchmarks and five speech frame rates, with additional evaluations on a larger backbone and alternative architectures. The main 1.7B experiments show improvements in average accuracy at every tested frame rate, with the largest gain of 14.05 percentage points at 25 Hz. The benefits are less consistent in the larger-model setting.

Critical strengths
  1. The method makes useful use of an existing property of parallel speech–text generation. The difference between speech and text rates creates positions that the model already processes but normally fills with padding. Using them to prepare the next spoken segment is a well-motivated design. The distinction from previous think-while-speaking methods is reasonably clear: THINKGAP explicitly targets existing decoding positions and constructs reasoning targets to fit their capacity. Keeping compression offline also makes the approach practical to incorporate into training.

  2. The main experimental improvement is substantial and supported by relevant controls. At 25 Hz, average accuracy increases from 54.68 to 68.73, with improvements on all five benchmarks. Alignment alone reaches 54.78, supporting the importance of adding reasoning content. Table 3 also shows a useful trade-off against full reasoning: THINKGAP achieves 68.73 versus 69.33 accuracy while reducing average forward steps from 297.3 to 235.7. These results make a convincing case that the proposed use of idle positions deserves further study.

Critical weaknesses
  1. The paper does not fully specify how the claimed budget constraint is enforced. Equation (15) permits negative carry-over, but does not explain how the final surplus is guaranteed to be nonnegative. Under the stated recurrence, an early segment can consume capacity that later spoken text needs. For example, with segment budgets c=(20,2)c=(20,2) and spoken-text lengths m=(2,10)m=(2,10), the total idle budget is 10, but the first segment can admit a reasoning block of length 18, including its markers. The final surplus is then negative. The implementation may contain a global repair step, as suggested by the appendix, but that step is not described.

    The inference procedure is similarly underspecified. Section 3.3 says the available budget depends on whether synthesis of the current sentence has finished, yet the mechanism for detecting completion and ending a reasoning block is unclear. This is particularly relevant to the Qwen2.5-Omni variant, where the thinker receives no speech-token feedback. These details are central to checking the claim that reasoning stays within the existing budget.

  2. The evaluation does not yet establish the claimed streaming behavior and its practical trade-offs. Table 4 measures the first speech token and total generation time. It does not explain when the first playable audio becomes available or whether playback waits between segments. The average timing results are encouraging, but they do not establish uninterrupted speech.

    There are also measurable changes that need to be characterized more carefully. At 25 Hz, average forward steps increase from 228.5 to 235.7; at 12.5 Hz, E2E latency increases from 8.17 to 8.65 seconds. These are modest differences, but no uncertainty estimates are provided. Meanwhile, average WER increases from 0.096 to 0.117, reaching 0.188 versus 0.100 at 5 Hz. Similar UTMOS scores do not resolve the text–speech consistency issue. The paper acknowledges these limitations, but its stronger statements about cost and uninterrupted generation go beyond the measurements. Missing training hyperparameters and exact decoding settings also make these comparisons difficult to reproduce.

Questions for authors
  1. What happens when Equation (15) produces a negative final surplus? Please specify the global correction or filtering rule, including whether previously inserted reasoning is removed or either stream is extended, and report how often this occurs.

  2. How is the end of a reasoning interval determined during inference? In particular, what happens when speech finishes before reasoning, and how is this handled in the Qwen2.5-Omni variant without speech-token feedback?

  3. Does the reported FST latency include audio decoding and buffering? Can the authors provide first-playable-audio latency and inter-segment waiting statistics from the evaluated setup? For the WER increase, does the difference persist after normalizing numerical expressions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Zjq5

We thank you for the valuable feedback. The major concerns you raised are covered in the General Response:

  • global budget correction and its frequency (G5),
  • the reasoning stopping rule and the Qwen2.5-Omni variant (G6),
  • first-playable-audio latency and FST/E2E latency with confidence intervals (G2, Tables R3–R5),
  • pauses in the generated audio and real-time factor (G3, Tables R6–R7),
  • uncertainty estimates (G1),
  • and WER after number normalization (G4).

Responses to the remaining questions follow below.


Q1. Negative final surplus of Equation (15).

The correction rule, its effect on both streams, and its frequency are given in G5: reasoning tokens are removed backwards from the last inserted block, neither the speech stream nor the spoken text is modified. Applied to your example, c = (20, 2), m = (2, 10):

  • Segment 1: a_1 = 20 − 2 + 0 = 18, b_1 = 16. A block of 16 reasoning tokens plus 2 markers is inserted, so s_1 = 0.
  • Segment 2 (final): a_2 = 2 − 10 + 0 = −8, so s_2 = −8. The text stream has 2 + 18 + 10 = 30 positions versus 22 speech slots.
  • Global correction removes 8 tokens from the end of block 1, leaving 8 reasoning tokens plus 2 markers. The text stream becomes 2 + 10 + 10 = 22 positions, matching the speech stream, and segment 2's text ends exactly with its speech.

Q2. Missing training hyperparameters and exact decoding settings.

The attachments provide the training and evaluation scripts, which contain the complete training hyperparameters and exact decoding settings for reproduction. We will also add the hyperparameters and decoding settings to the appendix.


Q3. Forward steps differences from Baseline.

ThinkGap reuses idle text slots and introduces no reasoning-only forward steps. For the same spoken output, the number of backbone steps is unchanged and depends only on speech duration (≈ duration × f_a). However, Baseline and ThinkGap may generate different spoken responses with different durations. The 'ThinkGap − Baseline' differences in Table 3 (+7.2, +3.0, +1.7, −0.6, and −1.0 steps) are equivalent to audio-duration differences of +0.29, +0.24, +0.20, −0.10, and −0.20 s, respectively. Thus, these step differences reflect differences in generated audio duration rather than additional steps introduced by reasoning tokens.

AuthorsOpenReview ↗

Response to Reviewer Zjq5

As the author–reviewer discussion period is about to end, we would like to kindly follow up. If our responses have addressed your concerns, we would appreciate it if you could consider updating your evaluation. If any concern remains, we would be glad to address it before the deadline.

Thank you again for your detailed review.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Global Response to All Reviewers

We thank all reviewers for their comments. To reduce duplication for the reviewers and the AC, we provide unified explanations in the General Response for similar questions raised by the reviewers.


G1. Additional Statistical Information

We followed the reviewers' suggestions. To more rigorously demonstrate the performance gains brought by ThinkGap, we supplemented the main experimental results with corresponding statistical information (mean and standard error), and estimated 95% confidence intervals via the bootstrap method for each metric.

Table R1: Accuracy (%; mean ± SE [95% CI], bootstrap with 10,000 resamples).

Method25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 Hz
Baseline54.68 ± 1.61 [51.51, 57.84]56.35 ± 1.57 [53.25, 59.43]56.70 ± 1.56 [53.63, 59.67]57.83 ± 1.57 [54.73, 60.84]57.05 ± 1.54 [54.05, 60.06]
Alignment54.78 ± 1.60 [51.61, 57.96]56.47 ± 1.59 [53.30, 59.59]54.71 ± 1.60 [51.58, 57.85]55.03 ± 1.61 [51.86, 58.14]57.90 ± 1.56 [54.86, 61.00]
ThinkGap68.73 ± 1.39 [66.03, 71.44]62.17 ± 1.52 [59.14, 65.11]60.73 ± 1.51 [57.72, 63.58]59.41 ± 1.55 [56.35, 62.36]61.71 ± 1.47 [58.78, 64.59]

Table R2: Paired accuracy improvement (percentage points; Δ [95% CI], paired bootstrap with 10,000 resamples).

Comparison25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 Hz
ThinkGap − Baseline+14.05 [10.62, 17.46]+5.82 [2.53, 9.18]+4.03 [1.02, 7.06]+1.57 [−1.35, 4.46]+4.66 [1.61, 7.69]
ThinkGap − Alignment+13.95 [10.41, 17.41]+5.70 [2.28, 9.16]+6.02 [3.01, 9.15]+4.37 [1.33, 7.46]+3.81 [0.84, 6.74]
Alignment − Baseline+0.10 [−2.41, 2.61]+0.12 [−2.41, 2.69]−1.99 [−4.76, 0.71]−2.80 [−5.70, 0.07]+0.85 [−2.06, 3.89]

The results presented in the tables above address the reviewers' concerns:

  1. (For Reviewer rhVf) There is no significant difference between Alignment and Baseline (all five CIs for 'Alignment − Baseline' include 0). However, the accuracy values of Alignment and Baseline are not identical. This is because although neither Alignment nor Baseline contains reasoning content, "no reasoning" does not imply "no change in computation". Alignment differs from Baseline in the following aspects: (i) Baseline maintains the same interaction paradigm as the pre-trained model, where all idle tokens are padded at the end of the generated text. In contrast, Alignment adopts a different interaction paradigm from the pre-trained model, which may lead to forgetting of pre-trained knowledge. (ii) Under Alignment's generation scheme, semantically equivalent text and speech segments are positioned closer to each other, which may facilitate generation. (iii) The layout distribution of the training data differs.

  2. (For all Reviewers) ThinkGap demonstrates statistically significant improvements over Baseline at all frame rates (all 95% CIs lie entirely above zero), with the exception of 6.25 Hz (where the CI encompasses zero). At lower frame rates (6.25 Hz and 5 Hz), the relative scarcity of text idle tokens reduces the computational space available for model reasoning. Consequently, the performance gains are insufficient to offset the degradation caused by the discrepancy between our interaction paradigm and the pre-training paradigm.

  3. Interestingly, since both Alignment and ThinkGap incur similar adaptation costs from paradigm shift during training, the benefit of replacing idle tokens with compressed reasoning tokens should be isolated by comparing ThinkGap against Alignment. This is indeed confirmed by our results: ThinkGap significantly outperforms Alignment across all frame rates (all CIs for 'ThinkGap − Alignment' exclude zero).


G2. Additional Latency Information

We agree that latency requires statistical analysis due to its high variance. We have added first-playable-audio (FPA) latency, measured from input readiness until the first streaming chunk is decoded and playable. To ensure rigor, we rerun all evaluations with warm-up (discarding the first run of each sample) and report FPA, FST (input readiness to first speech token, excluding decoding and buffering), and E2E (input readiness to complete waveform synthesis, including codec decoding and packetization).

Table R3: First-playable-audio latency (FPA) (seconds; mean ± SE [95% CI], bootstrap with 10,000 resamples).

Method25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 HzAvg
Baseline0.985 ± 0.001 [0.982, 0.987]0.889 ± 0.001 [0.887, 0.891]0.884 ± 0.001 [0.881, 0.886]0.954 ± 0.001 [0.952, 0.957]0.991 ± 0.001 [0.989, 0.993]0.941 ± 0.001 [0.939, 0.942]
Alignment0.984 ± 0.001 [0.982, 0.986]0.879 ± 0.001 [0.877, 0.881]0.883 ± 0.001 [0.880, 0.886]0.965 ± 0.001 [0.962, 0.967]0.988 ± 0.001 [0.985, 0.991]0.940 ± 0.001 [0.939, 0.941]
ThinkGap0.988 ± 0.001 [0.986, 0.991]0.885 ± 0.001 [0.883, 0.887]0.875 ± 0.001 [0.873, 0.878]0.961 ± 0.001 [0.959, 0.964]0.995 ± 0.001 [0.993, 0.997]0.941 ± 0.001 [0.940, 0.942]
ThinkGap − Baseline+0.004 ± 0.002 [+0.001, +0.007]−0.004 ± 0.002 [−0.007, −0.001]−0.008 ± 0.002 [−0.011, −0.005]+0.007 ± 0.002 [+0.004, +0.010]+0.004 ± 0.002 [+0.001, +0.007]+0.001 ± 0.001 [−0.001, +0.002]

Table R4: First speech token latency (FST) (seconds; mean ± SE [95% CI], bootstrap with 10,000 resamples).

Method25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 HzAvg
Baseline0.0965 ± 0.0002 [0.0961, 0.0969]0.1347 ± 0.0003 [0.1341, 0.1352]0.1709 ± 0.0005 [0.1700, 0.1719]0.2067 ± 0.0004 [0.2059, 0.2075]0.2431 ± 0.0005 [0.2422, 0.2441]0.1704 ± 0.0002 [0.1700, 0.1707]
Alignment0.0962 ± 0.0002 [0.0959, 0.0966]0.1332 ± 0.0003 [0.1327, 0.1337]0.1699 ± 0.0003 [0.1693, 0.1705]0.2089 ± 0.0004 [0.2082, 0.2098]0.2423 ± 0.0005 [0.2413, 0.2432]0.1701 ± 0.0002 [0.1698, 0.1705]
ThinkGap0.0973 ± 0.0003 [0.0968, 0.0978]0.1342 ± 0.0003 [0.1337, 0.1347]0.1691 ± 0.0004 [0.1684, 0.1699]0.2086 ± 0.0005 [0.2076, 0.2097]0.2436 ± 0.0003 [0.2430, 0.2443]0.1706 ± 0.0002 [0.1702, 0.1709]
ThinkGap − Baseline+0.0008 ± 0.0003 [+0.0001, +0.0014]−0.0004 ± 0.0004 [−0.0012, +0.0003]−0.0017 ± 0.0006 [−0.0030, −0.0006]+0.0020 ± 0.0006 [+0.0007, +0.0032]+0.0005 ± 0.0006 [−0.0006, +0.0016]+0.0002 ± 0.0002 [−0.0003, +0.0007]

Table R5: End-to-end latency (E2E) (seconds; mean ± SE [95% CI], bootstrap with 10,000 resamples).

Method25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 HzAvg
Baseline8.166 ± 0.217 [7.801, 8.650]6.361 ± 0.197 [6.114, 6.798]5.700 ± 0.039 [5.626, 5.780]6.113 ± 0.466 [5.408, 7.139]5.855 ± 0.381 [5.391, 6.682]6.439 ± 0.150 [6.183, 6.763]
Alignment8.028 ± 0.132 [7.834, 8.330]6.468 ± 0.134 [6.261, 6.774]9.555 ± 0.647 [8.370, 10.899]6.164 ± 0.346 [5.735, 6.928]5.387 ± 0.055 [5.290, 5.506]7.120 ± 0.155 [6.836, 7.444]
ThinkGap8.262 ± 0.215 [7.898, 8.733]6.882 ± 0.334 [6.305, 7.624]6.061 ± 0.266 [5.711, 6.666]5.637 ± 0.058 [5.532, 5.760]5.853 ± 0.392 [5.360, 6.714]6.539 ± 0.134 [6.307, 6.828]
ThinkGap − Baseline+0.096 ± 0.248 [−0.387, +0.585]+0.521 ± 0.389 [−0.222, +1.320]+0.361 ± 0.264 [+0.012, +0.967]−0.476 ± 0.467 [−1.516, +0.253]−0.001 ± 0.548 [−1.148, +1.138]+0.100 ± 0.187 [−0.271, +0.463]

As shown in the tables above, the Avg CIs of FPA, FST, and E2E for 'ThinkGap − Baseline' include zero, indicating no statistically significant average difference. Therefore, ThinkGap and Baseline have comparable latency.


G3. Continuity of Generated Speech

To demonstrate that ThinkGap does not introduce additional pauses into the generated audio, we analyze speech continuity at two levels: (i) algorithmic, ensuring ThinkGap does not generate excessive silence speech tokens due to text segmentation, and (ii) systemic, ensuring generation speed exceeds playback speed during streaming.

  1. Algorithmic analysis is critical for validating our approach. We conducted silence detection on all generated speech to identify any method-induced silent intervals.

    Specifically, we computed volume using 30 ms windows with a 10 ms hop and classified measurements below −40 dBFS (decibels relative to full scale) as silence. Continuous silence of ≥ 200 ms within speech was counted as a pause; leading and trailing silence was analyzed separately and not counted as pauses. Pause ratio = total pause duration / duration from the first non-silent frame to the last non-silent frame.

    Table R6: Pause statistics from silence detection.

    MethodPause ratio (%)Mean pause (s)Longest pause p50 / p95 (s)Files with ≥ 1 s pause (%)
    25.0 Hz
    Baseline19.30.470.38 / 0.890.3
    Alignment19.10.470.39 / 0.890.0
    ThinkGap19.10.470.38 / 0.890.2
    12.5 Hz
    Baseline18.90.480.38 / 0.890.2
    Alignment19.20.470.39 / 0.890.3
    ThinkGap19.00.480.38 / 0.900.2
    8.33 Hz
    Baseline19.00.480.38 / 0.890.3
    Alignment19.50.470.39 / 0.890.1
    ThinkGap18.90.490.38 / 0.900.2
    6.25 Hz
    Baseline18.70.530.38 / 0.890.2
    Alignment20.20.480.39 / 0.890.1
    ThinkGap18.90.480.38 / 0.890.1
    5.00 Hz
    Baseline19.30.520.39 / 0.900.1
    Alignment19.10.500.39 / 0.890.1
    ThinkGap19.20.490.38 / 0.900.1

    Results confirm that the three paradigms have similar pause ratios and mean pause durations. Thus, compared to the standard paradigm, ThinkGap does not show a clear increase in the measured acoustic pauses despite discontinuous text sequences.

  2. Systemic analysis. Uninterrupted streaming playback requires speech to be generated faster than it is played back, a requirement shared by the standard paradigm and ThinkGap. We computed the RTF (Real-Time Factor) of each sample as its generation time divided by its audio duration, and averaged it over samples. RTF < 1 indicates faster than real time.

    Table R7: Real-Time Factor on a single PPU-ZW810 (per-sample RTF; mean ± SE [95% CI], bootstrap with 10,000 resamples).

    Method25.0 Hz12.5 Hz8.33 Hz6.25 Hz5.00 Hz
    Baseline0.936 ± 0.001 [0.934, 0.938]0.729 ± 0.001 [0.726, 0.731]0.657 ± 0.001 [0.656, 0.659]0.620 ± 0.002 [0.617, 0.625]0.666 ± 0.049 [0.609, 0.772]
    ThinkGap0.989 ± 0.022 [0.952, 1.038]0.807 ± 0.044 [0.733, 0.908]0.694 ± 0.032 [0.656, 0.765]0.638 ± 0.005 [0.630, 0.648]0.676 ± 0.049 [0.617, 0.784]

    On a single PPU-ZW810, the mean RTFs of Baseline and ThinkGap are below 1 at all five frame rates, showing that speech is generated faster than it is played back, so streaming playback does not need to wait for generation at the system level. Since our improvements target the algorithmic level and gains are measured under identical system conditions, this serves as reference only, and optimizations such as vLLM, multi-GPU parallelism, or hardware upgrades can achieve better real-time performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.