- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors investigate why speculative decoding does not help in hierarchical speech generators that represent each frame with RVQ codes. They show that on Qwen3-omni, the residual codes in the RVQ codec, the slowdown comes from generating the residual codes of each drafted frame before its acceptance is known, since the target needs a complete frame as input to verify the next proposal, and show that feeding back only some of the tokens does not improve performance. They conclude that regular decoding is better in this setting.
Critical strengths
- The main problem the paper tries to tackle is important: Improving speculative decoding for speech models could be a useful approach for streaming TTS models.
- The analysis covers three models: Qwen3-Omni, Qwen3-TTS, and Moshi.
Critical weaknesses
-
Poor writing quality and presentation: The paper is very hard to follow. Despite reading it several times, it was very hard to understand what the paper is about and what the authors propose. It has many opaque sentences that use non-standard terminology without properly defining it. For instance, “Acoustic feedback” titles Section 3.3 and appears in the contributions, but it is never defined as a term. “Acoustic groups” appears in the abstract but is only explained later, in the Moshi-specific sense of a code group. The research narrative is also not clear enough and is hard to follow.
-
The major slowdown is not explained by the mechanism: The authors show that speculation is only 5% slower than regular decoding, but the end-to-end latency is about 50% worse. Therefore, most of the slowdown is due to implementation overhead.
-
Drafters are very weakly trained, which undermines the results: The authors train the draft models on only 200 prompts, and the joint-code drafters for just several epochs. Their poor residual overlap and the drop in acceptance after retraining look more like data and training limitations than evidence that the task is hard. Text drafters such as EAGLE are trained on far more data.
-
Limited evaluations and missing obvious metrics: The paper only evaluates on , with no tree drafting or longer drafts, and only evaluates WER, despite speaker similarity being a standard metric in speech generation models. Despite this limited setting, the paper overclaims that it explains “Why speculative decoding fails in hierarchical speech generation.”
Questions for authors
Can you explain what is Figure 1 showing?
What happens at K = 3? K=4?
Why not verify residuals exactly?
3 comments
Response to author rebuttal
I thank the authors for the rebuttal and the additional experiments. Unfortunately, I find that the issues in the manuscript are mostly unresolved, most importantly the presentation issues. Even after revision, the paper does not read as a polished research paper but as a collection of experiment reports. The main text contains many dense sequences of numbers without a clear narrative connecting the claim made, with evidence and conclusion. The structure appears driven by reviewer requests rather than a coherent argument. Experimental setups also vary across sections. Finally, the title claims speculative decoding “fails” while the authors’ own comment states they do not claim it is infeasible.
The rebuttal suffers from the same issues as the manuscript. For example, the response to Reviewer RbWr’s novelty concern immediately goes into technical details that are very hard to parse, rather than clearly stating the contribution. Addressing these issues would require a substantial rewrite beyond the scope of the rebuttal period. I therefore maintain my score.
In your follow-up comment, you noted the additional experiments we provided, and stated that you maintained your score because resolving the presentation issues (narrative flow, inline number density, varying setups across sections, and title scope) "would require a substantial rewrite beyond the scope of the rebuttal period."
We want to respond plainly, since none of the empirical findings in our revision are in dispute.
First, our contribution can be stated simply without technical notation. In multi-codebook speech models, predicting the next audio frame requires summing all codebooks of the current frame, so any cross-frame draft must fill in its remaining codebooks before later frames can be verified in parallel. Comparing six speech architectures shows that having multiple codebooks per frame is harmless when they are predicted in parallel (as in MiniCPM-O 2.6), and becomes a bottleneck only when residual codebooks are generated sequentially one by one (as in Qwen3-Omni, Moshi, and Qwen3-TTS), where preparing draft frames costs several times more GPU time than parallel verification saves. Furthermore, dropping fine codebooks from next-frame feedback breaks speech not because the input vector norm shrinks, but because omitted codebooks have a non-zero mean vector; adding their fixed offline mean back recovers nearly all of the lost transcription accuracy without training and points directly to how future speech backbones can be trained to verify from the primary code alone.
Second, our previous revision read densely because we tried to fold every reviewer-requested control directly into the earlier draft, and our comment distinguished unmodified sequential-RVQ models (where cross-frame speculation loses time) from parallel or modified designs (where it is viable). Rather than leaving any ambiguity or leaving the manuscript in that form, we rewrote the entire nine-page main text and appendices from scratch and uploaded a clean PDF. All inline statistical lists and ratio symbols have been moved out of the prose into Tables 1–3 and the appendix. The main text now follows one straight argument from architecture and three-GPU CUDA-Graph timing (Section 2), to shortcut controls on a unified 200-prompt benchmark (Section 3), to 10x drafter training on those same 200 prompts and hidden-state precision bounds (Section 4), and finally to model design (Section 5). We also narrowed the title to "Drafting Without Hearing: Why Cross-Frame Speculation Does Not Pay Off in Hierarchical Speech Generation" so there is no gap between title and claim.
Every technical question from the initial review has been answered with real-weight experiments, and the full rewrite you assumed could not be completed during the rebuttal period is now uploaded. We ask that the submission be judged on the manuscript as it now stands.
Clarified Mechanism, Drafter Controls, and Verification Alternatives
We thank Reviewer jfLy for the detailed questions and feedback. Below we address each point directly:
1. Terminology and Figure 1
We removed undefined shorthand (such as "acoustic groups") and defined primary codes, residual codes, and candidate completion at the start of the paper. Figure 1 compares what happens between drafting and parallel target verification across three setups:
2. Serving-level (~40%–50%) vs. isolated GPU slowdown
In live vLLM-Omni serving, Python dispatch and CPU-GPU stream synchronizations do account for most of the extra wall-clock delay above raw kernel time. However, isolating execution inside warmed CUDA Graphs (removing Python and CPU wait) shows speculative decoding remains slower on the GPU itself:
3. Are the results an artifact of training on 200 prompts?
We added three controls independent of the 200-prompt training set:
4. Deeper lookahead (3 or 4 frames) and tree drafting
Primary acceptance decays across positions—47.3% at pos 1, 27.0% at pos 2, and < 21% at pos 3—while each extra draft step adds another full 15-step serial preparation pass upfront. Standard tree drafting compounds this cost since each branch node requires its own 15-step completion before child nodes are verified.
5. Why not verify residual codes with exact rejection sampling?
Scoring 15 proposed residuals in one batched pass is fast (0.85 vs. 10.80 ms across 4 frames, 12.7x faster); the obstacle is proposing accurate 16-code frames cheaply:
6. Speaker similarity and deterministic ASR
Across all 256 saved H200 recordings, we added GE2E speaker similarity alongside deterministic Whisper (
temperature = 0) WER relative to paired ordinary audio. Feedback-only truncation (7 residuals, no drafter, full 16-code output) triples WER (11.7% to 35.2%) and drops speaker similarity from 1.000 to 0.808, while full-depth speculative runs keep speaker similarity at 0.901–0.913.