acceptodds
Under review as a conference paper at ICLR 2027

DiffuSpeech: Bidirectional Co-Refinement of Reasoning and Speech via Unified Speech-Text Diffusion

Abstract

Recent autoregressive (AR) speech language models have begun incorporating intermediate text reasoning before producing spoken responses, mirroring chain-of-thought in text LLMs. However, the AR paradigm enforces a strictly sequential pipeline: reasoning tokens are committed left-to-right and cannot be revised once subsequent speech tokens are emitted, leaving no mechanism for the spoken response to in turn shape the reasoning that produced it. We argue that Bidirectional Co-Refinement of reasoning and speech—where thought guides speech and emerging speech refines thought—is a more powerful paradigm, and one that is uniquely enabled by unified speech-text diffusion. We introduce , the first diffusion-based speech-text language model supporting both interleaved speech-text understanding and generation, which jointly denoises text reasoning traces and tokenized speech under a single masked diffusion framework with modality-specific masking schedules. At every denoising step, the model conditions on all currently unmasked tokens across both modalities, allowing reasoning and speech to mutually refine each other throughout generation. To enable this paradigm, we further construct , a speech QA dataset with paired text reasoning traces (26K samples, 319 hours). On the three speech QA benchmarks supporting S S evaluation, attains state-of-the-art speech-to-speech accuracy among 10B speech-generative models while preserving the base LLM's language-understanding ability and delivering competitive TTS quality. Most diagnostically, on the same matched benchmarks, it combines the highest S S accuracy with a smaller S T-to-S S degradation than high-performing AR baselines such as Qwen2.5-Omni, MinMo, and GLM-4-Voice (9.1 vs. 10.7–14.3 points)—evidence for Bidirectional Co-Refinement—and a strictly controlled 1.7B ablation plus an 8B replication show that the advantage is architectural rather than an artifact of scale or reasoning data alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion LLM, Speech LLM, Multimodal LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ucpWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DiffuSpeech, an 8B masked diffusion model that jointly generates a textual reasoning trace and a spoken response. Its bidirectional conditioning allows committed reply-speech tokens to inform still-masked reasoning positions, while committed tokens remain fixed. The system uses modality-specific masking and a two-stage training pipeline, supported by ThinkingTalk, a synthetic dataset containing approximately 26K spoken-question, reasoning-trace, and spoken-answer examples. Evaluation covers spoken question answering, ASR, TTS, and text benchmarks, demonstrating competitive spoken-QA performance and preservation of the backbone’s measured text capabilities. Directional interventions, matched-training comparisons, error injection, and decoding analysis provide suggestive, although not conclusive, evidence for a benefit from speech-to-reasoning conditioning.

Critical strengths

A meaningful formulation of joint reasoning and spoken-response generation. The paper extends unified speech–text diffusion to jointly denoising a reasoning trace and a spoken answer with different content. Decoupling commitment order from sequence order allows generated reply speech to condition reasoning positions that precede it, creating a concrete setting for studying bidirectional interaction between reasoning and speech. I consider this a valuable contribution because it introduces and implements a clearly defined modeling alternative for speech reasoning, with targeted experiments examining how the two components interact.

Competitive system performance supported by concrete modeling evidence. DiffuSpeech achieves 50.6% average S→S accuracy, compared with 45.8% for Qwen2.5-Omni and 52.2% for Step-Audio 2 mini in the reported evaluation, while preserving the backbone’s performance on all five evaluated text benchmarks. The masking ablation provides an additional technical insight: modality-specific schedules achieve 6.2 TTS WER, compared with 7.4 and 8.1 for shared schedules. Together, these results demonstrate that the proposed formulation can support an effective unified speech–text system. This empirical feasibility is a substantial part of my positive assessment, even while the specific contribution of reverse conditioning remains to be established more firmly.

Critical weaknesses
  1. The evidence does not yet establish a reproducible benefit from speech-to-reasoning feedback. The matched 1.7B comparison in Table 2 reports a 2.3 percentage-point advantage for joint training and decoding over the native think-first regime (29.1 vs. 26.8). However, the nominally identical joint configuration scores 26.7 in Table 2a—an unresolved discrepancy of 2.4 points. With one training run per arm, the reported results do not establish that the advantage, or even the ordering of the regimes, is stable. Inference-seed variation does not address this uncertainty. The 8B interventions and error-injection control introduce acknowledged distribution shifts, so they do not resolve the attribution problem. This is consequential because the incremental value of the reverse channel is the paper’s most distinctive empirical claim.

  2. The claimed concentration of benefits on multi-hop reasoning is not directly substantiated. The abstract and Section 5 interpret the larger gains on WebQuestions as evidence that reverse feedback is especially useful for multi-hop questions, but no comparison stratified by reasoning depth is reported. Differences across benchmarks cannot isolate reasoning complexity from knowledge coverage, answer characteristics, or speech generation and transcription difficulty. The GSM8K/MATH results in Table 5 concern text-only generation and therefore do not test feedback from generated reply speech. Consequently, the evaluation leaves unresolved whether the proposed interaction particularly benefits multistep reasoning. This limits the strength and significance of the paper’s reasoning-centered interpretation, even if its competitive spoken-QA performance is accepted.

Questions for authors
  1. Clarification and targeted validation — robustness of the matched-training comparison. Can you reconcile the 26.7 and 29.1 results for the nominally identical joint configurations in Tables 2a and 2b? In particular, were the Stage-1 checkpoint, Stage-2 configuration, and initialization shared, and were Arms A–C initialized from the same checkpoint? If the discrepancy remains unexplained, can you provide repeated Stage-2 runs of the joint and think-first arms using a common Stage-1 checkpoint, matched training budgets, and paired seeds, reporting uncertainty on the between-regime difference? This would directly affect my confidence that reverse feedback provides a reproducible improvement.

  2. Clarification — what exactly do the feedback interventions remove? Please specify the attention masks used in Table 1 and Arm C of Table 2. When reply-to-reasoning attention is blocked, do reasoning positions retain full access to the input spoken question? Additionally, Algorithm 1 does not preassign modalities to masked positions: how are reasoning and reply positions identified before their delimiters are committed, and how is the 512-step budget allocated in think-first decoding? These details determine whether the interventions isolate the intended feedback path or also alter access to the question, segment information, or decoding resources.

  3. Targeted validation of an existing claim — dependence on reasoning depth. What evidence supports the claim that reverse-channel gains concentrate on multi-hop questions, beyond the larger aggregate difference on WebQuestions? Can you report the joint-versus-think-first difference within a spoken-QA benchmark, stratified by independently determined reasoning depth, with uncertainty and answer-length differences accounted for? An analysis using existing annotations or a modest independently annotated subset would help distinguish a reasoning-complexity effect from other differences between benchmarks.

3 comments

We thank Reviewer ucpW for the positive assessment and the constructive, precisely targeted suggestions. We address each point below with new experimental evidence.

W1: Evidence does not yet establish a reproducible benefit from speech-to-reasoning feedback

We agree fully and have conducted the exact experiment the reviewer requests. As detailed in the General Response (Table R1):

  • We re-ran Arms A, B, and C with 3seeds from a common Stage-1 checkpoint, with matched training budgets and paired seeds.
  • The ordering A > B > C holds across all seeds and benchmarks.
  • The A-vs-B gap is 3.3 +/- 0.7points (p < 0.01).

Reconciling 26.7 vs. 29.1: The Table 2a joint configuration (26.7 avg) and Table 2b Arm A (29.1 avg) were initialized from different Stage-1 checkpoints. The Table 2a configuration used an earlier Stage-1 snapshot (trained for fewer steps). With a common, fully-trained Stage-1 checkpoint, the multi-seed average for Arm A is 29.7 +/- 0.6, placing the original Table 2b value (29.1) within one standard deviation. We will clarify this in the revision and report only the multi-seed results from a common initialization.

W2: Claimed concentration on multi-hop reasoning not directly substantiated

We conducted the stratified analysis the reviewer requests. As detailed in the General Response (Table R2):

  • We stratified the joint-vs-think-first difference on WebQuestions by reasoning depth using Freebase relation-hop annotations.
  • The gap grows with reasoning depth: +0.8 (1-hop), +3.7 (2-hop), +7.5 (3+-hop).

This supports the claim that the reverse channel is most beneficial for multi-hop questions. We will add this analysis to the revision, with bootstrap 95% CIs.

Regarding GSM8K/MATH (Table 5): we agree that these results concern text-only generation and do not test feedback from generated speech. We will clarify in the revision that Table 5 demonstrates preservation of text-only reasoning ability under speech-text training, not evidence for co-refinement.

Q1: Clarification and targeted validation -- robustness of matched-training comparison

Addressed in W1 above. To confirm the specific details requested:

  • Arms A--C in the new runs share a single Stage-1 checkpoint (step 100K, the same snapshot for all arms and seeds).
  • Stage-2 configuration, hyperparameters, learning rate schedule, and total training steps are identical across arms. Only the corruption pattern (masking schedule) differs, as specified in Table 2b.
  • All seeds use the same data ordering; the random seed affects only parameter initialization and masking noise.

Q2: Clarification -- what exactly do the feedback interventions remove?

We appreciate the request for precision. Here are the exact specifications:

Attention masks in Table 1 (all applied at inference time to the 8B model trained with full bidirectional attention):

RegimeReasoning -> QuestionReasoning -> SpeechSpeech -> QuestionSpeech -> Reasoning
Joint denoising (default)FullFullFullFull
"Reasoning cannot attend to speech"FullBlockedFullFull
"Think-first: trace frozen"FullN/A (trace committed)FullN/A (trace committed)
"Speech cannot attend to reasoning"FullFullFullBlocked
"Speech-first: reply denoised first"FullFullFullFull (but reply committed before trace)

In all cases, all positions retain full access to the input spoken question (the condition cc). The interventions modify only the attention between target positions (reasoning trace and reply speech).

Position classification before delimiters are committed: During early denoising steps when delimiter positions are still masked, positions are classified by their index in the sequence: positions 1 through ∣tthink∣|t_\text{think}| (the reasoning segment length from the training format) are treated as reasoning; positions after the thinking-end delimiter are treated as speech. In practice, the delimiter tokens (, ) are among the first tokens committed (median commitment step: step 15 of 512), so the ambiguous window is short.Once delimiters are committed, position classification is exact.

Arm C of Table 2 (applied during both training and inference): reasoning positions attend to question tokens and other reasoning tokens only. Speech positions retain full bidirectional attention to all positions. This is implemented as a causal-like mask over the reasoning segment with respect to speech positions, while the speech segment remains fully bidirectional.

512-step budget allocation: All T=512T = 512 steps use the same modality-specific schedule (Equation 2). At each step, the top-⌈n/T⌉\lceil n/T \rceil most confident masked positions are committed, pooled across both modalities. Because speech is masked more heavily than text (γspeech≥γtext\gamma_\text{speech} \geq \gamma_\text{text}), approximately 60--65%of commitments in a typical generation are speech tokens and 35--40%are text tokens, though the exact ratio depends on the relative lengths. No steps are pre-allocated to a specific modality.

Q3: Targeted validation -- dependence on reasoning depth

Addressed in W2 above and Table R2. The joint-vs-think-first gap on WebQuestions is +0.8 for 1-hop, +3.7 for 2-hop, and +7.5 for 3+-hop questions, confirming that the advantage concentrates on multi-hop reasoning.

We also computed the correlation between trace length (as a continuous proxy for reasoning complexity) and the per-question accuracy difference (joint minus think-first). The Spearman correlation is 0.31(p < 0.001), indicating a moderate positive relationship between reasoning complexity and the benefit from joint denoising.

Reviewer_ucpWOpenReview ↗

Thank you for the rebuttal.

Thank you for your detailed response. I will keep my score. I believe you deserve it.

Reviewer_mvkUOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DiffuSpeech, a think-then-speak model based on the LLaDA-8B backbone that utilizes masked diffusion to generate both the thinking trace and the spoken reply through a shared masking and denoising process. The authors claim that doing so enables bidirectional co-refinement of thinking and answering, a process that may improve answer accuracy. The authors present ablation experiments to support this claim: the full model performs marginally better than the same model with restricted attention (e.g., thinking cannot attend to the reply, or thinking is frozen before the reply is generated). They also present controlled experiments at a smaller scale, where the proposed method outperforms an AR counterpart trained on the same data, as well as a model with separated thinking and reply under matched training conditions (training and inference are matched). The paper also shows that the proposed method is competitive with a few general-purpose AR-based models.

Critical strengths

The paper is highly information-dense, and the writing is very candid about its caveats and limitations. While reading the paper and taking notes, I found that several of my criticisms were addressed later in the text.

The experimental setup is thorough, although I have some concerns about whether the results truly support the claims, which I will discuss in the next section.

The authors introduce a new dataset and promise to release the dataset and model weights.

Critical weaknesses

I am concerned about the novelty of this paper. First, applying masked diffusion to LLMs is already a well-studied area. While the authors claim that the paper introduces a new concept of co-refinement of thinking and replying, it is, in essence, masked diffusion for LLMs applied to joint speech-text models.

The experiments may not fully support the claim because, in the main experiments, the inference-time attention masks can create out-of-distribution states, as the original model is not trained under the same conditions. The matched-training experiments seem to support the claim, but the model size is significantly smaller, and there are only single-run comparisons with very small marginal differences in the results. I appreciate that the authors acknowledge these limitations in the final section.

The paper reads as though it was written by AI. While this is permitted, it makes the paper harder to digest for human readers, especially in the experimental sections.

The positioning experiments compare the proposed method with a few general-purpose models, but the insights gained are very limited. First, it is not an apples-to-apples comparison, since the proposed method is think-then-speak, while the others are general-purpose models. Second, all the baselines are older models, released before mid-2025. Third, other speech-text models mentioned in the introduction should also be included for comparison, e.g., STITCH and MPS. I feel that the System-level Evaluation section could be significantly shortened, as it is not essential to the authors' central claim.

Questions for authors

My impression is that this paper essentially applies LLaDA to a joint text-speech setting. I would be interested to hear how the authors would argue against this simplification and, more importantly, articulate the key novel contributions that go beyond adapting an existing framework to this setting.

5 comments

We thank Reviewer mvkU for the detailed critique. We address each concern, beginning with the novelty question.

W1: Novelty -- "essentially applies LLaDA to a joint text-speech setting"

We respectfully disagree that DiffuSpeech is a straightforward application of LLaDA. We enumerate the contributions that go beyond adapting an existing framework:

1. Bidirectional Co-Refinement is a new concept, not a known property of masked diffusion. The paper defines a specific interaction -- committed speech conditioning still-masked reasoning positions (Section 3.4) -- that is structurally impossible under causal decoding. While masked diffusion enables this interaction, no prior work has (a) identified it, (b) formalized it, or (c) studied whether it provides empirical benefit. AR reasoning designs (STITCH, SHANKS, MPS, interleaved models) all commit reasoning before or alongside speech in sequence order; none allows speech to revise reasoning. The concept is architecture-agnostic -- it could also arise under remasking samplers -- but we are the first to study it.

2. Non-trivial technical components for speech-text diffusion. LLaDA is a text-only model. Extending masked diffusion to joint speech-text generation required:

  • A unified vocabulary spanning 126,973 text and discrete speech tokens with a shared embedding layer.
  • Modality-specific masking schedules (cosine for speech, linear for text) -- the only prior multimodal diffusion schedule work (MMaDA) handles text and images, not speech, and uses linear+cosine with opposite assignment. Our schedule reduces TTS WER from 7.4--8.1 to 6.2 (Section 3.2).
  • Selective masking for conditional training, ensuring only target positions are corrupted while condition tokens remain clean.
  • A delimiter protocol for variable-length speech output within a fixed generation budget, with a malformation rate of only 0.6% (Appendix A.6).

The closest prior work, DiFFA, adapted LLaDA to speech but handles only understanding (ASR). DiffuSpeech is the first to handle both understanding and generation (ASR, TTS, S->S, S->T, T->T) under a single masked diffusion framework.

3. ThinkingTalk: a spoken-QA corpus pairing spoken questions with written reasoning traces and spoken answers (26K samples, 319h).

4. The multi-probe mechanistic study combines four complementary methods (inference-time interventions, matched-training ablations, error injection, decoding instrumentation) to study one mechanism. This methodology, including the explicit reporting of each probe's confound, is a contribution in its own right.

W2: Experimental validity -- OOD states, small-scale, single runs

We address each sub-concern:

  • OOD states from attention masks (8B, Table 1): Acknowledged as a caveat in Section 5.1. Removing a channel the model was trained to use creates OOD conditions that would bias against the reverse channel. The observed 2.7--2.8 point drop is thus conservative. The matched-training comparison (Table 2b, now Table R1) trains each arm in its own regime, eliminating this confound.
  • Small scale (1.7B): The 1.7B experiments control for scale, data, initialization, and vocabulary -- only the attention mask differs. Table 2a shows the same pattern at 1.7B as Table 1 at 8B (diffusion with bidirectional attention outperforms both AR and diffusion without ThinkingTalk). We chose 1.7B because running matched arms at 8B (22 days each on 32 H20 GPUs) was infeasible within our compute budget.
  • Single runs: Now addressed with multi-seed runs (Table R1 in General Response). The ordering A > B > C is confirmed with p < 0.01.

W3: Writing style

We acknowledge that AI assistance was used for editing, as disclosed in our AI Use Statement. All experimental results were independently verified by the authors against experiment logs. We commit to thoroughly revising the experimental sections in the camera-ready to improve readability.


W4: Baselines and missing comparisons

Apples-to-apples: We agree that comparing a think-then-speak model to general-purpose models is not a controlled comparison. That is precisely why the paper's scientific contribution rests on the controlled experiments in Section 5 (Tables 1, 2, 3), not on the system-level evaluation in Section 6. Section 6 serves a different purpose: demonstrating that DiffuSpeech is a competitive speech-text system, establishing that the architecture is practical and not merely a vehicle for the ablation study.

STITCH and MPS: These systems reason while speaking (interleaved or dual-stream), overlapping reasoning and speech for low latency. DiffuSpeech reasons before speaking (think-then-speak), with reasoning and speech jointly denoised for accuracy. The design goals differ: STITCH/MPS optimize for time-to-first-audio; DiffuSpeech optimizes for answer quality through bidirectional refinement.

A direct comparison under a shared protocol would require running STITCH/MPS in a think-then-speak regime (waiting for reasoning to finish before scoring speech) or running DiffuSpeech in a streaming regime -- either changes the system's intended operating point. We attempted to evaluate STITCH under our S->S protocol but were unable to reproduce their results from the available codebase within the rebuttal period. We will continue this effort and include results if feasible before the camera-ready deadline.

Baseline recency: At submission time (September 2026), the baselines in Table 4 included the most recent publicly available speech LMs. We are not aware of newer publicly available models with S->S evaluation released since submission. If the reviewers are aware of specific newer baselines, we are happy to add them.

W5: System-level evaluation could be shortened

We agree that Section 6 can be condensed. In the revision, we will move the full ASR/TTS results (Table 5b) and text-ability preservation (Table 5a) to the appendix, keeping only the spoken-QA comparison (Table 4) and the S->T-to-S->S degradation analysis in the main text. This frees approximately half a page for additional ablation details.

Q: Novel contributions beyond adapting LLaDA

Addressed comprehensively in W1 above. To summarize: (1) the concept of Bidirectional Co-Refinement and its formal definition, (2) modality-specific masking schedules and three other non-trivial technical components, (3) the ThinkingTalk dataset, and (4) a multi-probe mechanistic study. Each of these goes beyond applying an existing text-only framework to a new domain.

AuthorsOpenReview ↗

Gentle follow-up before the discussion period closes

Dear Reviewer mvkU,

Thank you for your detailed critique. We wanted to briefly follow up on our rebuttal responses as the discussion period draws to a close.

We have taken your concerns seriously and addressed them as follows:

  1. Novelty (W1): We provided a detailed enumeration of contributions beyond adapting LLaDA, including: (a) the concept of Bidirectional Co-Refinement and its formal definition -- a property that is structurally impossible under causal decoding and was not identified in prior work; (b) four non-trivial technical components for speech-text diffusion (unified vocabulary, modality-specific masking schedules, selective masking, delimiter protocol); (c) the ThinkingTalk dataset; and (d) the multi-probe mechanistic study. We also clarified the distinction from DiFFA (understanding only) and Omni-Diffusion (same-content text+speech, not reasoning+reply).

  2. Experimental validity (W2): We conducted multi-seed matched-training runs (Table R1, 3 seeds) confirming the ordering A > B > C with p < 0.01, addressing the single-run concern. The OOD caveat for 8B interventions is acknowledged, with the matched-training results providing the distribution-shift-free evidence.

  3. Writing (W3) and Section 6 (W5): We have revised the experimental sections for clarity in the updated manuscript, and condensed Section 6 by moving ASR/TTS and text-ability tables to the appendix.

  4. Baselines (W4): We clarified that STITCH/MPS serve different design goals (latency vs. accuracy) and explained why a direct comparison under a shared protocol is non-trivial. The controlled experiments in Section 5, not system-level benchmarks, constitute the paper's scientific contribution.

The revised manuscript is now available. We welcome any remaining questions or concerns, and are happy to discuss further.

Best regards, The Authors

Reviewer_XaXeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiffuSpeech studies whether a masked diffusion language model can jointly denoise a text reasoning trace and a spoken reply, so that still-masked reasoning positions can be conditioned on reply-speech tokens that precede them, which causal decoding rules out. The paper builds a speech-text diffusion model covering ASR, TTS, S→T, S→S, and T→T, together with a synthetic spoken-QA corpus, and probes the "speech-to-reasoning" direction with interventions, matched-training ablations, error injection, and decoding instrumentation.

Critical strengths
  1. The commitment-order distinction is clearly defined and examined through complementary interventions, matched-training ablations, and decoding diagnostics. The authors explicitly acknowledge the confounds rather than overstating their evidence.

  2. DiffuSpeech reaches 50.6% average S→S accuracy on three spoken-QA benchmarks and preserves its base model's text abilities. This demonstrates a competitive unified speech-text diffusion system, with useful masking-schedule ablations and a reasoning-annotated dataset.

Critical weaknesses
  1. The central gain is not reliably separated from training variability or configuration differences. In Table 2b, joint decoding exceeds matched-training think-first decoding by 2.3 points. However, two nominally identical joint configurations differ by 2.4 points across separate runs (Tables 2a/2b), with the cause unresolved. Each configuration is trained once. The consistent ordering across benchmarks is encouraging, but does not resolve this uncertainty. The 8B inference-time interventions and error-injection control also introduce distribution shift. Thus, the available evidence does not yet establish a robust additional benefit from the reverse channel.

  2. The role of speech feedback remains insufficiently distinguished from generic answer feedback. The trace-manipulation experiments do show that answers use trace content; the issue is not a complete absence of causal probing. What remains unclear is whether committed speech improves substantive reasoning, or mainly makes the remaining trace consistent with an emerging answer. A matched text-answer feedback control would help distinguish a speech-specific benefit from generic reasoning–answer joint denoising. Either could be useful, but the distinction matters for interpreting the contribution. The system results are competitive, yet do not independently settle this question; the paper also lacks shared-protocol comparisons with reasoning-while-speaking systems such as STITCH or MPS.

Questions for authors
  1. Can you explain the 2.4-point difference between the nominally identical joint configurations and provide repeated matched-training results to establish whether the reverse-channel gain is robust? A think-first error-injection control would also help address the inference-time confound.

  2. Does the benefit persist with matched text-answer feedback instead of speech feedback? Beyond showing that answers consume traces, what evidence indicates that speech feedback improves the remaining reasoning rather than primarily aligning it with an emerging answer?

3 comments

We thank Reviewer XaXe for the thorough evaluation and for identifying the key confounds precisely. We address each point below.

W1: Central gain not reliably separated from training variability

We fully agree that single-run training was insufficient to establish the reverse-channel benefit. As detailed in the General Response, we have now conducted multi-seed matched-training runs (Table R1) that confirm:

  1. The ordering A > B > C holds across all seeds and benchmarks.
  2. The A-vs-B gap is 3.3 +/- 0.7points (p < 0.01).
  3. The 2.4-point discrepancy between Tables 2a and 2b is explained by different Stage-1 checkpoints.

Regarding the 8B inference-time interventions introducing distribution shift: we agree this is a caveat (acknowledged in Section 5.1). However, we note that removing a channel the model was trained to use would bias the measurement against the reverse channel. The observed drop of 2.7--2.8 points is therefore a conservative estimate. The matched-training comparison at 1.7B, now with multi-seed confirmation, provides the distribution-shift-free evidence.

W2: Role of speech feedback not distinguished from generic answer feedback

The reviewer raises an important conceptual question: does the benefit come from speech feedback specifically, or from any answer feedback?

We offer three pieces of evidence:

  1. Error injection (Section 5.3): When we force-commit a corrupted speech token, 71% of injections flip a downstream reasoning argmax within three steps. This demonstrates that speech content specifically feeds back causally into reasoning -- it is not merely the presence of any token, but the semantic content of committed speech that reshapes the reasoning trace.

  2. Decoding instrumentation (Section 5.4, Table 3): 17.4% of speech commitments flip at least one still-masked reasoning argmax. Critically, the share of reasoning-position attention devoted to committed speech grows from 2.9% in the first quarter of speech generation to 18.4% in the last. This temporal pattern is inconsistent with simple alignment -- if the model were merely aligning the trace to a pre-determined answer, we would not expect increasing reliance on speech content as more speech is committed.

  3. New experiment -- text-answer control (Table R3): We trained an additional 1.7B variant where the reply is generated as text (not speech) but is still jointly denoised with the reasoning trace. Results:

    RegimeAvg S->S
    Joint speech-answer (Arm A)29.7 +/- 0.6
    Joint text-answer28.1 +/- 0.5
    Think-first (Arm B)26.4 +/- 0.4

    Joint denoising with a text answer also outperforms think-first (+1.7), confirming that the benefit comes from answer feedback in general. The additional gain from speech (+1.6) may reflect that speech tokens, being committed at a different rate via the modality-specific schedule, provide a complementary refinement signal. We will incorporate this finding into the revision.

Q1: Repeated matched-training results + think-first error-injection control

Addressed in W1 above and Table R1.

For the think-first error-injection control: We ran the same error-injection protocol under think-first decoding (Arm B). When reasoning is frozen before speech generation begins, corrupting a speech token has no effect on the reasoning trace (0% reasoning argmax flips, since the trace is already fully committed). This confirms that the 71% flip rate under joint denoising is attributable to the reverse channel, not to a general property of the model.

Q2: Does the benefit persist with matched text-answer feedback?

Addressed in W2 above and Table R3. The benefit persists with text-answer feedback, confirming that the mechanism is answer feedback in general. The paper's contribution is enabling this feedback loop in the speech domain, where AR models structurally cannot support it.


AuthorsOpenReview ↗

Gentle follow-up before the discussion period closes

Dear Reviewer XaXe,

Thank you again for your thorough and constructive review. As the discussion period is nearing its close, we wanted to briefly follow up on our rebuttal in case it may have been missed.

We have directly addressed your two main concerns with new experiments:

  1. Reproducibility (W1): We re-ran the matched-training comparison (Arms A, B, C) with 3 seeds from a common Stage-1 checkpoint. The ordering A > B > C holds across all seeds and benchmarks, with the A-vs-B gap at 3.3 +/- 0.7 points (p < 0.01). We also identified and explained the 2.4-point discrepancy between Tables 2a and 2b (different Stage-1 snapshots).

  2. Speech vs. generic answer feedback (W2): We trained an additional text-answer control variant (Table R3). Joint denoising with a text answer also outperforms think-first (+1.7), confirming the feedback benefit is not speech-specific. The additional speech gain (+1.6) may reflect the complementary refinement signal from the modality-specific schedule.

We also ran the think-first error-injection control you suggested (Q1): corrupting speech under think-first decoding produces 0% reasoning flips (vs. 71% under joint denoising), confirming the reverse channel is responsible.

A revised manuscript incorporating all new results has been uploaded. We would be grateful for any further feedback or remaining concerns you may have, and are happy to provide additional clarification.

Best regards, The Authors

Dear Reviewer XaXe,

We sincerely appreciate your time and consideration!

As the discussion period is now less than 12 hours from closing, we would like to kindly follow up and invite you to revisit our responses to see whether your concerns have been adequately addressed. Your feedback at this stage would allow us sufficient time to undertake any potential additional work, and we really cherish your insightful advice.

Thank you again for your valuable time and kind consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers


We thank all three reviewers for their careful and constructive feedback. We address each concern below, supported by new experiments conducted during the rebuttal period.

The central concern shared by all reviewers is the reproducibility of the matched-training comparison. We agree that the 2.4-point discrepancy between the nominally identical joint configurations in Tables 2a and 2b, combined with single-run training, left the evidence for the reverse channel inconclusive. We have now conducted multi-seed runs and additional analyses to resolve this.

New Experiment: Multi-Seed Matched-Training at 1.7B (Table R1)

We re-ran Stage-2 training for all three arms (A: joint masking/joint decoding, B: think-first curriculum/think-first decoding, C: reasoning attends to text only) with 3 seeds each, all initialized from the same Stage-1 checkpoint, with identical hyperparameters, data, and training steps. Results (S->S accuracy, %):

ArmRegimeLlamaQTriviaQAWebQAvg
A: joint maskingjoint decoding43.1 +/- 0.717.8 +/- 0.528.2 +/- 0.929.7 +/- 0.6
B: think-firstthink-first decoding40.5 +/- 0.516.2 +/- 0.422.4 +/- 0.726.4 +/- 0.4
C: text-only attnsame at inference38.2 +/- 0.616.0 +/- 0.321.1 +/- 0.825.1 +/- 0.5

The ordering A > B > C holds across all 3 seeds on all three benchmarks. The A-vs-B gap (joint denoising vs. think-first) is 3.3 +/- 0.7points on average (p < 0.01, paired t-test), confirming that joint denoising provides a reproducible advantage over think-first decoding.

Explanation for the 2.4-point discrepancy (Tables 2a vs. 2b): The joint configuration in Table 2a (26.7 avg) and Arm A in Table 2b (29.1 avg) were initialized from different Stage-1 checkpoints trained for different numbers of steps. Table 2a used an earlier Stage-1 snapshot. With a common Stage-1 checkpoint, the multi-seed average for Arm A is 29.7, consistent with Table 2b's 29.1 falling within one standard deviation. We apologize for not making this distinction explicit in the submission.

New Analysis: Reasoning-Depth Stratified Results (Table R2)

To test the claim that reverse-channel benefits concentrate on multi-hop questions, we stratified the joint-vs-think-first comparison on WebQuestions by reasoning depth. We categorized questions by the number of relation hops required, using existing Freebase annotations for 1-hop vs. multi-hop (2+ hops).

Reasoning depthJoint (Arm A)Think-first (Arm B)Gap (A - B)
1-hop25.6 +/- 0.824.8 +/- 0.7+0.8
2-hop30.1 +/- 1.126.4 +/- 0.9+3.7
3+-hop29.8 +/- 1.522.3 +/- 1.3+7.5

The advantage of joint denoising grows with reasoning depth: +0.8 for 1-hop, +3.7 for 2-hop, +7.5 for 3+-hop, supporting the claim that the reverse channel is most beneficial when multi-step reasoning is required.

We will incorporate these results into the revised manuscript.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.