- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents DiffuSpeech, an 8B masked diffusion model that jointly generates a textual reasoning trace and a spoken response. Its bidirectional conditioning allows committed reply-speech tokens to inform still-masked reasoning positions, while committed tokens remain fixed. The system uses modality-specific masking and a two-stage training pipeline, supported by ThinkingTalk, a synthetic dataset containing approximately 26K spoken-question, reasoning-trace, and spoken-answer examples. Evaluation covers spoken question answering, ASR, TTS, and text benchmarks, demonstrating competitive spoken-QA performance and preservation of the backbone’s measured text capabilities. Directional interventions, matched-training comparisons, error injection, and decoding analysis provide suggestive, although not conclusive, evidence for a benefit from speech-to-reasoning conditioning.
Critical strengths
A meaningful formulation of joint reasoning and spoken-response generation. The paper extends unified speech–text diffusion to jointly denoising a reasoning trace and a spoken answer with different content. Decoupling commitment order from sequence order allows generated reply speech to condition reasoning positions that precede it, creating a concrete setting for studying bidirectional interaction between reasoning and speech. I consider this a valuable contribution because it introduces and implements a clearly defined modeling alternative for speech reasoning, with targeted experiments examining how the two components interact.
Competitive system performance supported by concrete modeling evidence. DiffuSpeech achieves 50.6% average S→S accuracy, compared with 45.8% for Qwen2.5-Omni and 52.2% for Step-Audio 2 mini in the reported evaluation, while preserving the backbone’s performance on all five evaluated text benchmarks. The masking ablation provides an additional technical insight: modality-specific schedules achieve 6.2 TTS WER, compared with 7.4 and 8.1 for shared schedules. Together, these results demonstrate that the proposed formulation can support an effective unified speech–text system. This empirical feasibility is a substantial part of my positive assessment, even while the specific contribution of reverse conditioning remains to be established more firmly.
Critical weaknesses
-
The evidence does not yet establish a reproducible benefit from speech-to-reasoning feedback. The matched 1.7B comparison in Table 2 reports a 2.3 percentage-point advantage for joint training and decoding over the native think-first regime (29.1 vs. 26.8). However, the nominally identical joint configuration scores 26.7 in Table 2a—an unresolved discrepancy of 2.4 points. With one training run per arm, the reported results do not establish that the advantage, or even the ordering of the regimes, is stable. Inference-seed variation does not address this uncertainty. The 8B interventions and error-injection control introduce acknowledged distribution shifts, so they do not resolve the attribution problem. This is consequential because the incremental value of the reverse channel is the paper’s most distinctive empirical claim.
-
The claimed concentration of benefits on multi-hop reasoning is not directly substantiated. The abstract and Section 5 interpret the larger gains on WebQuestions as evidence that reverse feedback is especially useful for multi-hop questions, but no comparison stratified by reasoning depth is reported. Differences across benchmarks cannot isolate reasoning complexity from knowledge coverage, answer characteristics, or speech generation and transcription difficulty. The GSM8K/MATH results in Table 5 concern text-only generation and therefore do not test feedback from generated reply speech. Consequently, the evaluation leaves unresolved whether the proposed interaction particularly benefits multistep reasoning. This limits the strength and significance of the paper’s reasoning-centered interpretation, even if its competitive spoken-QA performance is accepted.
Questions for authors
-
Clarification and targeted validation — robustness of the matched-training comparison. Can you reconcile the 26.7 and 29.1 results for the nominally identical joint configurations in Tables 2a and 2b? In particular, were the Stage-1 checkpoint, Stage-2 configuration, and initialization shared, and were Arms A–C initialized from the same checkpoint? If the discrepancy remains unexplained, can you provide repeated Stage-2 runs of the joint and think-first arms using a common Stage-1 checkpoint, matched training budgets, and paired seeds, reporting uncertainty on the between-regime difference? This would directly affect my confidence that reverse feedback provides a reproducible improvement.
-
Clarification — what exactly do the feedback interventions remove? Please specify the attention masks used in Table 1 and Arm C of Table 2. When reply-to-reasoning attention is blocked, do reasoning positions retain full access to the input spoken question? Additionally, Algorithm 1 does not preassign modalities to masked positions: how are reasoning and reply positions identified before their delimiters are committed, and how is the 512-step budget allocated in think-first decoding? These details determine whether the interventions isolate the intended feedback path or also alter access to the question, segment information, or decoding resources.
-
Targeted validation of an existing claim — dependence on reasoning depth. What evidence supports the claim that reverse-channel gains concentrate on multi-hop questions, beyond the larger aggregate difference on WebQuestions? Can you report the joint-versus-think-first difference within a spoken-QA benchmark, stratified by independently determined reasoning depth, with uncertainty and answer-length differences accounted for? An analysis using existing annotations or a modest independently annotated subset would help distinguish a reasoning-complexity effect from other differences between benchmarks.
3 comments
Q2: Clarification -- what exactly do the feedback interventions remove?
We appreciate the request for precision. Here are the exact specifications:
Attention masks in Table 1 (all applied at inference time to the 8B model trained with full bidirectional attention):
| Regime | Reasoning -> Question | Reasoning -> Speech | Speech -> Question | Speech -> Reasoning |
|---|---|---|---|---|
| Joint denoising (default) | Full | Full | Full | Full |
| "Reasoning cannot attend to speech" | Full | Blocked | Full | Full |
| "Think-first: trace frozen" | Full | N/A (trace committed) | Full | N/A (trace committed) |
| "Speech cannot attend to reasoning" | Full | Full | Full | Blocked |
| "Speech-first: reply denoised first" | Full | Full | Full | Full (but reply committed before trace) |
In all cases, all positions retain full access to the input spoken question (the condition ). The interventions modify only the attention between target positions (reasoning trace and reply speech).
Position classification before delimiters are committed: During early denoising steps when delimiter positions are still masked, positions are classified by their index in the sequence: positions 1 through (the reasoning segment length from the training format) are treated as reasoning; positions after the thinking-end delimiter are treated as speech. In practice, the delimiter tokens (, ) are among the first tokens committed (median commitment step: step 15 of 512), so the ambiguous window is short.Once delimiters are committed, position classification is exact.
Arm C of Table 2 (applied during both training and inference): reasoning positions attend to question tokens and other reasoning tokens only. Speech positions retain full bidirectional attention to all positions. This is implemented as a causal-like mask over the reasoning segment with respect to speech positions, while the speech segment remains fully bidirectional.
512-step budget allocation: All steps use the same modality-specific schedule (Equation 2). At each step, the top- most confident masked positions are committed, pooled across both modalities. Because speech is masked more heavily than text (), approximately 60--65%of commitments in a typical generation are speech tokens and 35--40%are text tokens, though the exact ratio depends on the relative lengths. No steps are pre-allocated to a specific modality.
Q3: Targeted validation -- dependence on reasoning depth
Addressed in W2 above and Table R2. The joint-vs-think-first gap on WebQuestions is +0.8 for 1-hop, +3.7 for 2-hop, and +7.5 for 3+-hop questions, confirming that the advantage concentrates on multi-hop reasoning.
We also computed the correlation between trace length (as a continuous proxy for reasoning complexity) and the per-question accuracy difference (joint minus think-first). The Spearman correlation is 0.31(p < 0.001), indicating a moderate positive relationship between reasoning complexity and the benefit from joint denoising.
Thank you for the rebuttal.
Thank you for your detailed response. I will keep my score. I believe you deserve it.
We thank Reviewer ucpW for the positive assessment and the constructive, precisely targeted suggestions. We address each point below with new experimental evidence.
W1: Evidence does not yet establish a reproducible benefit from speech-to-reasoning feedback
We agree fully and have conducted the exact experiment the reviewer requests. As detailed in the General Response (Table R1):
Reconciling 26.7 vs. 29.1: The Table 2a joint configuration (26.7 avg) and Table 2b Arm A (29.1 avg) were initialized from different Stage-1 checkpoints. The Table 2a configuration used an earlier Stage-1 snapshot (trained for fewer steps). With a common, fully-trained Stage-1 checkpoint, the multi-seed average for Arm A is 29.7 +/- 0.6, placing the original Table 2b value (29.1) within one standard deviation. We will clarify this in the revision and report only the multi-seed results from a common initialization.
W2: Claimed concentration on multi-hop reasoning not directly substantiated
We conducted the stratified analysis the reviewer requests. As detailed in the General Response (Table R2):
This supports the claim that the reverse channel is most beneficial for multi-hop questions. We will add this analysis to the revision, with bootstrap 95% CIs.
Regarding GSM8K/MATH (Table 5): we agree that these results concern text-only generation and do not test feedback from generated speech. We will clarify in the revision that Table 5 demonstrates preservation of text-only reasoning ability under speech-text training, not evidence for co-refinement.
Q1: Clarification and targeted validation -- robustness of matched-training comparison
Addressed in W1 above. To confirm the specific details requested: