acceptodds
Under review as a conference paper at ICLR 2027

Drafting Without Hearing: Why Speculative Decoding Fails in Hierarchical Speech Generation

Abstract

Speech models such as Qwen3-Omni represent each audio frame with a primary code followed by residual codes that refine its acoustic detail. The completed frame becomes part of the history used to predict the next frame. Speculative decoding offers a natural way to accelerate this process by verifying future proposals in parallel. On frozen Qwen3-Omni, scoring three positions costs only 23% more than scoring one, yet complete speculative generation is consistently slower than ordinary decoding. We trace the lost verification saving to generating each draft's remaining acoustic codes, a step we call acoustic completion. Those codes are needed to verify later proposals before the draft's acceptance is known. Preparing these inputs and completing output frames account for 81.7% of measured speculative model computation. Even with a free drafter, the remaining computation nearly matches ordinary decoding at the observed acceptance rate. Truncating acoustic feedback to reduce this cost distorts subsequent target predictions. A controlled intervention raises word error rate from 11.68% to 40.97% even though every output frame retains all its codes. Drafting complete acoustic groups transfers the same prediction task to the drafter. On Moshi, more accurate complete-group drafters increase the number of accepted code groups, yet their candidate-generation cost still prevents a speedup. These results identify acoustic completion as a cost that higher proposal accuracy alone does not eliminate: a draft may be cheap to verify, yet expensive to make verifiable. Under the conditions we study, ordinary decoding is the better policy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, speech generation, audio generation, multimodal language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_jfLyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate why speculative decoding does not help in hierarchical speech generators that represent each frame with RVQ codes. They show that on Qwen3-omni, the residual codes in the RVQ codec, the slowdown comes from generating the residual codes of each drafted frame before its acceptance is known, since the target needs a complete frame as input to verify the next proposal, and show that feeding back only some of the tokens does not improve performance. They conclude that regular decoding is better in this setting.

Critical strengths
  • The main problem the paper tries to tackle is important: Improving speculative decoding for speech models could be a useful approach for streaming TTS models.
  • The analysis covers three models: Qwen3-Omni, Qwen3-TTS, and Moshi.
Critical weaknesses
  • Poor writing quality and presentation: The paper is very hard to follow. Despite reading it several times, it was very hard to understand what the paper is about and what the authors propose. It has many opaque sentences that use non-standard terminology without properly defining it. For instance, “Acoustic feedback” titles Section 3.3 and appears in the contributions, but it is never defined as a term. “Acoustic groups” appears in the abstract but is only explained later, in the Moshi-specific sense of a code group. The research narrative is also not clear enough and is hard to follow.

  • The major slowdown is not explained by the mechanism: The authors show that speculation is only 5% slower than regular decoding, but the end-to-end latency is about 50% worse. Therefore, most of the slowdown is due to implementation overhead.

  • Drafters are very weakly trained, which undermines the results: The authors train the draft models on only 200 prompts, and the joint-code drafters for just several epochs. Their poor residual overlap and the drop in acceptance after retraining look more like data and training limitations than evidence that the task is hard. Text drafters such as EAGLE are trained on far more data.

  • Limited evaluations and missing obvious metrics: The paper only evaluates on k=2k=2, with no tree drafting or longer drafts, and only evaluates WER, despite speaker similarity being a standard metric in speech generation models. Despite this limited setting, the paper overclaims that it explains “Why speculative decoding fails in hierarchical speech generation.”

Questions for authors

Can you explain what is Figure 1 showing?

What happens at K = 3? K=4?

Why not verify residuals exactly?

3 comments
AuthorsOpenReview ↗

Clarified Mechanism, Drafter Controls, and Verification Alternatives

We thank Reviewer jfLy for the detailed questions and feedback. Below we address each point directly:

1. Terminology and Figure 1

We removed undefined shorthand (such as "acoustic groups") and defined primary codes, residual codes, and candidate completion at the start of the paper. Figure 1 compares what happens between drafting and parallel target verification across three setups:

  • (a) Text & single-codebook speech: Drafted tokens are already complete inputs, verified in one parallel pass with zero preparation cost.
  • (b) Parallel-head multi-codebook (MiniCPM-O 2.6): Four linear heads predict Frame 1's four codebooks simultaneously in 0.07 ms (2.6% of step time) to form Position 2's input, preserving the verification saving.
  • (c) Sequential RVQ speech (Qwen3-Omni): Position 2 is blocked until a residual predictor generates 15 dependent codes one by one (5.24 ms, 80.2% of frame time), spending more GPU time preparing candidates than parallel verification saves. (Panels b–c show positions 1–2 of a two-draft round and omit position 3 for clarity.)

2. Serving-level (~40%–50%) vs. isolated GPU slowdown

In live vLLM-Omni serving, Python dispatch and CPU-GPU stream synchronizations do account for most of the extra wall-clock delay above raw kernel time. However, isolating execution inside warmed CUDA Graphs (removing Python and CPU wait) shows speculative decoding remains slower on the GPU itself:

  • 7-step truncated preparation: Halving residual steps degrades WER severely (29.5%), yet isolated GPU time is still 18%–26% slower than ordinary decoding on H200 (7.73 vs. 6.55 ms/frame, or 7.39 ms even with a zero-cost drafter).
  • Full 15-step preparation: Isolated GPU execution runs 1.53x–2.65x slower (11.17 vs. 6.55 ms/frame in Figure 2b), whereas MiniCPM-O 2.6 in the same CUDA-Graph harness completes frames in 2.6% of step time—confirming an architectural GPU bottleneck.

3. Are the results an artifact of training on 200 prompts?

We added three controls independent of the 200-prompt training set:

  1. Pretrained target prefixes: Reusing Qwen3-Omni's pretrained Talker layers (1 to 19 of 20 layers, no small-data limit) lifts progress to 2.15 frames/round at 16/20 layers and restores WER (9.6% vs. 9.0%), yet runs 1.53x slower due to draft cost (a 20/20 exact check at 100% acceptance runs 1.12x slower).
  2. Hidden-state interpolation (448 utterances): Blending exact target states with predicted states shows WER recovers within 20% state error, whereas 15-step break-even (2.94 of 3 frames/round) requires ~2% error even for a free predictor.
  3. On-policy replay audit: Continuous-AdamW retraining with replay improves visited rollout states (1.74 to 1.96), while held-out progress drops (1.78 to 1.63) as unverified draft residuals conflict with clean target states.

4. Deeper lookahead (3 or 4 frames) and tree drafting

Primary acceptance decays across positions—47.3% at pos 1, 27.0% at pos 2, and < 21% at pos 3—while each extra draft step adds another full 15-step serial preparation pass upfront. Standard tree drafting compounds this cost since each branch node requires its own 15-step completion before child nodes are verified.

5. Why not verify residual codes with exact rejection sampling?

Scoring 15 proposed residuals in one batched pass is fast (0.85 vs. 10.80 ms across 4 frames, 12.7x faster); the obstacle is proposing accurate 16-code frames cheaply:

  • Parallel 16-code proposals: Three trained 16-head parallel drafters (77M–93M) reach only 16%–25% per-codebook overlap without sequential conditioning, driving exact whole-frame acceptance below 0.03%.
  • Sequential residual proposals: Generating 15 residuals serially from the draft state already pays full preparation upfront; exact rejection under the target state then forces serial resampling from the first rejected depth and invalidates later draft frames whenever any residual changes. On Moshi, exact whole-frame self-speculation runs 1.11x–1.90x slower.

6. Speaker similarity and deterministic ASR

Across all 256 saved H200 recordings, we added GE2E speaker similarity alongside deterministic Whisper (temperature = 0) WER relative to paired ordinary audio. Feedback-only truncation (7 residuals, no drafter, full 16-code output) triples WER (11.7% to 35.2%) and drops speaker similarity from 1.000 to 0.808, while full-depth speculative runs keep speaker similarity at 0.901–0.913.

Reviewer_jfLyOpenReview ↗

Response to author rebuttal

I thank the authors for the rebuttal and the additional experiments. Unfortunately, I find that the issues in the manuscript are mostly unresolved, most importantly the presentation issues. Even after revision, the paper does not read as a polished research paper but as a collection of experiment reports. The main text contains many dense sequences of numbers without a clear narrative connecting the claim made, with evidence and conclusion. The structure appears driven by reviewer requests rather than a coherent argument. Experimental setups also vary across sections. Finally, the title claims speculative decoding “fails” while the authors’ own comment states they do not claim it is infeasible.

The rebuttal suffers from the same issues as the manuscript. For example, the response to Reviewer RbWr’s novelty concern immediately goes into technical details that are very hard to parse, rather than clearly stating the contribution. Addressing these issues would require a substantial rewrite beyond the scope of the rebuttal period. I therefore maintain my score.

In your follow-up comment, you noted the additional experiments we provided, and stated that you maintained your score because resolving the presentation issues (narrative flow, inline number density, varying setups across sections, and title scope) "would require a substantial rewrite beyond the scope of the rebuttal period."

We want to respond plainly, since none of the empirical findings in our revision are in dispute.

First, our contribution can be stated simply without technical notation. In multi-codebook speech models, predicting the next audio frame requires summing all codebooks of the current frame, so any cross-frame draft must fill in its remaining codebooks before later frames can be verified in parallel. Comparing six speech architectures shows that having multiple codebooks per frame is harmless when they are predicted in parallel (as in MiniCPM-O 2.6), and becomes a bottleneck only when residual codebooks are generated sequentially one by one (as in Qwen3-Omni, Moshi, and Qwen3-TTS), where preparing draft frames costs several times more GPU time than parallel verification saves. Furthermore, dropping fine codebooks from next-frame feedback breaks speech not because the input vector norm shrinks, but because omitted codebooks have a non-zero mean vector; adding their fixed offline mean back recovers nearly all of the lost transcription accuracy without training and points directly to how future speech backbones can be trained to verify from the primary code alone.

Second, our previous revision read densely because we tried to fold every reviewer-requested control directly into the earlier draft, and our comment distinguished unmodified sequential-RVQ models (where cross-frame speculation loses time) from parallel or modified designs (where it is viable). Rather than leaving any ambiguity or leaving the manuscript in that form, we rewrote the entire nine-page main text and appendices from scratch and uploaded a clean PDF. All inline statistical lists and ratio symbols have been moved out of the prose into Tables 1–3 and the appendix. The main text now follows one straight argument from architecture and three-GPU CUDA-Graph timing (Section 2), to shortcut controls on a unified 200-prompt benchmark (Section 3), to 10x drafter training on those same 200 prompts and hidden-state precision bounds (Section 4), and finally to model design (Section 5). We also narrowed the title to "Drafting Without Hearing: Why Cross-Frame Speculation Does Not Pay Off in Hierarchical Speech Generation" so there is no gap between title and claim.

Every technical question from the initial review has been answered with real-weight experiments, and the full rewrite you assumed could not be completed during the rebuttal period is now uploaded. We ask that the submission be judged on the manuscript as it now stands.

Reviewer_RbWrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the failure of speculative decoding to accelerate hierarchical speech generation. The authors identify acoustic completion, which generates the residual acoustic codes required for verifying future proposals, as a major computational bottleneck. Experiments on Qwen3-Omni, Moshi, and Qwen3-TTS show that the additional computation largely offsets the benefits of parallel verification. The authors also find that incomplete acoustic feedback degrades speech content and that more accurate complete-group drafting still fails to achieve acceleration.

Critical strengths
  1. The paper investigates a practically relevant problem concerning inference efficiency in hierarchical speech generation.
  2. The computational profiling provides a breakdown of the costs associated with drafting, verification, and acoustic completion.
  3. The experiments offer empirical observations about the impact of incomplete acoustic feedback on subsequent speech generation.
Critical weaknesses

I find this paper somewhat incomplete, particularly in terms of its short introduction and limited experimental evaluation. My main concerns are as follows:

  1. Limited novelty. The main finding is that generating additional acoustic codes introduces computational overhead that can offset the benefits of speculative decoding. However, I do not see a clear methodological contribution or sufficiently general theoretical insight beyond empirical profiling of existing models.

  2. Insufficient introduction. I find the Introduction unusually short for a conference paper. It does not clearly establish the research gap, discuss the limitations of existing methods, or explain why the identified problem deserves further investigation.

  3. Limited experiments. The authors evaluate only 32 English reading prompts for Qwen3-Omni, four questions for Moshi, and a small computational study for Qwen3-TTS. I do not think such limited experiments are sufficient to support the paper's broad conclusions about hierarchical speech generation.

  4. Unclear research significance. Although the authors identify acoustic completion as a computational bottleneck, I find the practical implications rather limited. The paper does not offer much insight into how this bottleneck could be addressed, and I am not convinced that the findings extend beyond the specific implementations being evaluated.

Overall, I feel that the paper mainly identifies a computational bottleneck in several existing implementations, without providing sufficiently novel or generalizable insights. In its current form, I do not think the work is mature enough for ICLR.

Questions for authors

None. See weaknesses.

1 comment
AuthorsOpenReview ↗

Structural Contribution, Generality, and Architectural Implications

Thank you for the feedback on novelty, experimental scale, and practical implications. We rewrote the entire nine-page manuscript from scratch in plain English and added substantial new experiments so the paper provides general architectural findings and concrete design guidance rather than a profiling log of one setup.

1. Generalizable insight beyond profiling existing implementations Two findings extend across hierarchical speech generation (Sections 2–3):

  • Why multi-codebook inputs are not the cause: In multi-codebook speech models, predicting the next frame requires summing all codebooks of the current frame, so any drafted frame must fill in its remaining codebooks before future frames can be verified in parallel. Comparing six speech architectures (0.5B to 9B) shows that summing multiple codebooks per frame is harmless when those codebooks are predicted in parallel: MiniCPM-O 2.6 (8B) sums four codebooks at every step, predicts all four simultaneously through parallel linear heads in under 3% of step time, and preserves the parallel verification saving. Cross-frame speculation loses time only when residual codebooks are generated sequentially one by one (Qwen3-Omni, Moshi, Qwen3-TTS), which forces every unverified draft through serial single-token GPU kernels costing four to six times more time than parallel verification saves.
  • Why truncating residual codebooks breaks speech: When omitting the last eight residual codebooks from next-frame feedback to cut preparation cost, word error rate jumps from 9.7% to 34.2%. A natural guess is that summing half as many embeddings shrinks the input vector's norm—yet rescaling the truncated vector back to the full 16-codebook norm actually causes runaway speech (134.9% word error rate). Instead, we find that fine residual codebooks have non-zero mean vectors in embedding space: dropping them subtracts a constant baseline offset expected by the temporal backbone. Adding back the precomputed offline average vector of each omitted codebook recovers 92% of the word-error-rate penalty with zero training and zero runtime cost, isolating the remaining serial steps as a purely structural cost barrier.

2. Substantially expanded experimental evaluation To support general conclusions well beyond our initial 32-prompt setup:

  • Expanded our speech-quality and causal-intervention benchmark more than sixfold to 200 speaker-disjoint LibriSpeech prompts, replicated across two GPU architectures and scored with both deterministic Whisper transcription and speaker similarity (Section 3).
  • Expanded our drafter training set tenfold from 200 to 2,000 LibriSpeech trajectories and evaluated online across all 200 test prompts, alongside 448 controlled hidden-state interpolation runs and 17 draft configurations spanning Qwen3-Omni and Moshi (Section 4).
  • Replicated isolated CUDA-Graph latency and profiler measurements across three GPU generations (Ampere A800, Blackwell RTX PRO 6000, and Hopper H200) across four multi-codebook models (Section 2).

3. Expanded Introduction, prior-work synthesis, and actionable model design We expanded the Introduction to two full pages (with a new schematic comparing single-stream, parallel-head, and sequential-RVQ verification) and added a dedicated section (Section 5) reconciling prior speech acceleration papers and laying out two concrete ways to build speculation-friendly streaming speech models:

  • Parallel multi-head intra-frame readouts: Replacing sequential residual predictors with parallel output heads eliminates preparation overhead while retaining multi-codebook representations, as MiniCPM-O 2.6 already demonstrates at 8B scale.
  • Depth-dropout training with static codebook means: For models that keep sequential RVQ for compression efficiency, randomly dropping a suffix of residual codebooks from next-frame feedback during training—while adding the static mean vectors of omitted codebooks uncovered in Section 3—trains the temporal backbone to verify future frames conditioned on the first codebook alone, deferring residual generation until after primary tokens are accepted.
Reviewer_ADWDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper conducted a systematic analysis on why speculative decoding SD fails on speech generation tasks. Specifically, they show that the main overhead is preparing candidate inputs and finishing output frames, incomplete acoustic feedback can lead to quality degradation and parallel prediction of codes still fails to improve wall-clock latency.

Critical strengths

Study why SD fail on speech generation is an interesting topic and its insight could inspire future works on accelerating speech generation

Critical weaknesses
  1. The paper claims the main overhead is on the input preparing and completing output, however the specific overhead is not examined nor studied. The experiment is only conducted on a specific hardware H200, which have high compute power, it may well be the case that for lower-end hardware the operation becomes compute bound, and a speed-up may be observed since the cost of model forward would be relatively more expensive compared with prepartion on this hardware.
  2. The experiment lacks detailed analysis and breakdown of the alleged overhead. Are they bounded by CPU operations, memory latency, or bandwidth?
  3. In general, the presentation of paper is hard to follow, considerable revision is needed to meet the acceptance criteria
  4. Many experiment design choices are not discussed. One example is the size of drafter. Will scaling drafter increase the number of accepted tokens and hence increase speed?
  5. The base model size is also not discussed.
Questions for authors

See weakness. I do not have questions whose answers could materially affect my assessment.

1 comment
AuthorsOpenReview ↗

GPU Profiling, Hardware Generality, and Scaling Controls

Thank you for raising these concrete questions on CPU vs. GPU attribution, hardware tiers, base-model size, and drafter scaling. We ran new experiments to test each one directly and rewrote the nine-page manuscript so the argument reads cleanly without dense notation.

1. Is the bottleneck caused by CPU overhead, memory bandwidth, or serial GPU execution?

We traced both live vLLM-Omni serving and isolated static CUDA Graphs using NVIDIA Nsight Systems (Section 2). In live serving, CPU-GPU stream synchronizations do account for most of the extra wall-clock delay above raw kernel time. However, when we capture the temporal backbone, the residual predictor, and the draft model inside static CUDA Graphs—removing Python dispatch and CPU wait completely—speculative decoding still runs slower on the GPU than ordinary decoding, even if we set the draft model's own runtime to zero.

Profiling inside the CUDA Graph explains why a residual predictor 23x smaller than the temporal backbone takes four times longer per frame on the GPU: generating 15 residual codes one after another launches 540 tiny single-token matrix-vector kernels per frame. Because each step cannot look up its input embedding until the previous step samples a discrete code, these microsecond-scale kernels cannot overlap; their small thread grids underfill the GPU and reach less than one-third of peak memory bandwidth. Scoring all 15 residual codes at once in a single batched pass using the exact same weights runs over 12x faster, confirming that serial step-by-step dependence—not CPU sync or saturated memory bandwidth—is the device bottleneck.

2. Does lower-end GPU hardware become compute-bound and produce a speedup?

We tested your hypothesis directly by running the same static CUDA-Graph benchmark across three GPU generations: Ampere A800 (lower compute and HBM2e bandwidth), Blackwell RTX PRO 6000 (GDDR7), and Hopper H200 (HBM3e). Moving to slower hardware scales the temporal backbone and the residual predictor together rather than making single-token residual steps compute-bound: on all three GPUs, Qwen3-Omni spends roughly 80% of ordinary frame time inside its 15-step residual predictor (and Qwen3-TTS spends roughly 72%), so preparing candidate frames still costs four to six times more GPU time than parallel verification saves (Section 2).

3. How do base-model size and architecture change the cost balance?

Comparing six open-weight speech models from 0.5B to 9B (Section 2) shows that model size shifts the break-even threshold, while intra-frame structure decides whether speculation pays off. For instance, Moshi uses a much larger 7.7B dense temporal backbone that takes 68% of frame time, lowering the required acceptance rate compared with Qwen3-Omni's 3.3B sparse Mixture-of-Experts backbone—yet Moshi's sequential 8-step residual chain still consumes enough of the parallel saving that self-speculation runs slower than ordinary decoding. Conversely, MiniCPM-O 2.6 (8B) also sums four audio codebooks into every step, but predicts all four simultaneously through parallel output heads in 2.6% of step time, leaving the parallel verification saving intact.

4. Does scaling up the drafter (model size or training data) yield a speedup?

We tested both larger drafters and 10x more training data (Section 4):

  • Larger drafter capacity: Reusing the target model's own early layers as self-speculative drafters (up to 19 of Qwen3-Omni's 20 layers, and up to 30 of Moshi's 32 layers) raises token acceptance, but draft latency rises alongside accuracy, leaving every approximate predictor 1.5x to 2.6x slower than ordinary decoding.
  • 10x training data: Expanding our learned drafter's training set from 200 to 2,000 LibriSpeech trajectories and evaluating online across 200 speaker-disjoint prompts lifts average progress from 1.77 to 2.03 committed frames per round (out of a maximum of 3), still well short of the 2.94 frames per round needed to break even on full 15-step preparation. Controlled hidden-state interpolation across 448 utterances shows why: speech quality recovers within 20% hidden-state error, whereas latency break-even requires roughly 2% error even for a zero-cost drafter.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Revision Overview

Based on the review and discussion feedback, we rewrote all nine pages of the main text and the appendices from scratch in plain English and uploaded a clean unhighlighted PDF under the updated title "Drafting Without Hearing: Why Cross-Frame Speculation Does Not Pay Off in Hierarchical Speech Generation". All inline statistical lists and ratio symbols have been moved out of the prose into three summary tables and the appendix.

The revised paper establishes four results that form one continuous argument:

  1. Sequential intra-frame generation, not multiple codebooks, causes the slowdown (Section 2). Multi-codebook speech models sum all codebooks of the current frame before predicting the next frame, so every drafted frame must fill in its remaining codebooks before later frames can be verified in parallel. Comparing six open-weight models shows that this is harmless when a model predicts all codebooks of a frame simultaneously (MiniCPM-O 2.6 finishes all four codebooks in under 3% of step time, keeping parallel verification profitable), and breaks cross-frame speedup only when residual codebooks are generated one after another (Qwen3-Omni, Moshi, Qwen3-TTS). Across Ampere A800, Blackwell RTX PRO 6000, and Hopper H200 inside static CUDA Graphs with zero CPU wait, those serial single-token GPU kernels cost four to six times more time than parallel verification saves.

  2. Truncating residual feedback is a quality problem we can fix, but a serial-cost problem we cannot (Section 3). Across 200 speaker-disjoint LibriSpeech prompts on two GPUs, omitting the last eight residual codebooks from next-frame feedback degrades transcription accuracy not because the input vector has smaller norm (rescaling norms triggers runaway speech), but because omitted fine codebooks have a non-zero average embedding. Adding their fixed offline mean vector back recovers 92% of the word-error-rate penalty without training, yet the remaining seven serial residual steps per draft still cost several times more than verification saves. Drafting all 16 codebooks in parallel drops exact whole-frame acceptance below 0.03%.

  3. Stronger or better-trained drafters face a tenfold precision gap (Section 4). Blending exact and predicted hidden states across 448 utterances shows that speech quality returns to baseline within 20% hidden-state error, whereas latency break-even requires roughly 2% error even if the draft model itself were free. Training our drafter on 10x more data (2,000 LibriSpeech trajectories, tested online across all 200 test prompts) or reusing up to 19 of the target model's 20 layers improves token acceptance, but every approximate drafter still runs at least 1.5x slower than ordinary decoding.

  4. Design implications for future streaming speech models (Section 5). These results explain why reported speech speedups rely on single-codebook tokenizers, relaxed verification, or intra-frame speculation, and point to two speculation-friendly designs: parallel multi-head readouts, or training sequential-RVQ backbones with depth dropout plus static omitted-codebook mean vectors so future frames can be verified from the first codebook alone.

AuthorsOpenReview ↗

On the Scientific Value of Structural Negative Results

We would like to emphasize an important aspect of our contribution that may be obscured by its negative speedup outcome.

A well-substantiated negative result is not merely a failed attempt at acceleration. When it identifies a structural bottleneck, rules out plausible confounding explanations through controlled experiments, and establishes the quantitative conditions under which an approach becomes uneconomical, it provides actionable knowledge for the research community.

Our work does not claim that speculative decoding is inherently infeasible for speech generation. Rather, it identifies the specific architectural dependencies that undermine its benefits in sequential RVQ speech generators. Together, our three-generation GPU profiling, training-independent drafter controls, and parallel-head structural counterexample (MiniCPM-O 2.6) separate this architectural bottleneck from serving implementation overhead or insufficient drafter training.

As open-source multimodal serving systems (such as vLLM-Omni) seek to port text-style speculation onto streaming speech models, such findings help prevent repeated investment in unproductive post-hoc drafter scaling, conserve substantial engineering effort and GPU resources, and redirect research toward speculation-friendly speech architectures from the outset.

We believe that establishing why a seemingly promising optimization fails—and precisely when it need not fail—is itself a meaningful scientific contribution.

We respectfully ask the Reviewers and Area Chair to evaluate this work not solely by whether it achieves a positive speedup, but by the explanatory power, experimental evidence, and practical implications of its structural diagnosis.

I don't want to argue with the authors. All the reviewers rated it 1 on the initial submission. This paper was actually an AI slop when it was submitted, with only 7 pages of incomplete main content.

This reflects the authors' disrespectful attitude toward ICLR, including reviewers, ACs, SACs, PCs, and all the readers.

The revised version looks better, but the contribution's significance is minor. The main problem is that the paper concludes that cross-frame speculative decoding is ineffective for hierarchical speech generation. However, the experiments only evaluate a limited set of drafting and verification strategies. In particular, potentially more efficient approaches, such as batched candidate completion and combining intra-frame with cross-frame speculation, are not sufficiently explored. Therefore, the results demonstrate the limitations of the evaluated implementations rather than establishing a fundamental architectural bottleneck.

I will keep my score. This is my final decision.

But I will respect the AC's decision.

We would like to make one final clarification for both Reviewer RbWr and the Area Chair.

We acknowledge that our initial submission had shortcomings and that substantial revisions do not automatically warrant acceptance. However, labeling the manuscript "AI slop" and attributing a "disrespectful attitude toward ICLR" to its authors goes beyond scientific criticism. Our AI assistance was transparently disclosed, and we respectfully question whether such personal characterizations are consistent with ICLR's expectations for professional and constructive reviewing.

Regarding the two technical objections, both are already addressed in the revised manuscript:

  • Combining intra-frame and cross-frame speculation: Section 5 analyzes this direction and explains why accelerating intra-frame generation does not eliminate the cross-frame completion bottleneck.
  • Batched candidate completion: Sections 2–3 and Appendix B.1 distinguish operations that can be batched from the sequential dependencies involved in generating candidate frames.

We do not claim to have exhausted every possible optimization. However, these directions have not been overlooked. ICLR's Reviewer Guidelines expect reviewers to engage seriously with author responses and revised evidence, and to explain what has or has not changed their assessment. Maintaining a negative recommendation is entirely legitimate, but its scientific justification should accurately reflect the revised manuscript.

We respect the reviewer's right to maintain their score. Nevertheless, given the Area Chair's responsibility to oversee review quality and maintain professional discussions, we respectfully ask the AC to examine whether the remaining objections are technically substantiated and whether the review itself meets the conference's standards for constructive and respectful evaluation, seeking further clarification or intervening as appropriate.

We recognize that the final decision may remain negative, and we are not requesting acceptance on the basis of our revisions. We simply ask for a fair, evidence-based assessment of the scientific contribution, separate from personal characterizations of the authors.

We leave the final evaluation to the Area Chair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.