acceptodds
Under review as a conference paper at ICLR 2027

Training Reshapes Width-Dependent Acceptance in Block-Diffusion Speculative Decoding

Abstract

Inference-time width selection balances a drafter's acceptance against execution cost. We study how training changes that balance. In block-diffusion drafting, a narrower forward can lower acceptance even when the verified prefix is unchanged. A three-arm experiment separates training forward width from the positions included in the loss. With identical prefix supervision, narrow-layout training improves acceptance at narrow evaluation widths and lowers it at a wider width in all three paired seeds. On 900 prompts from six sources, the gain extends to width 8, the configuration selected for serving. This finding motivates cost-driven adaptation: identify promising training layouts from measured costs, adapt with standard cross-entropy, then select each head's deployment width independently. On MI308X, both supervision-matched heads select width 8 from the same candidates. Narrow-layout training improves throughput by 1.58% on average, with positive gains in all three seeds and sessions and 17 of 18 paired comparisons during nine-minute runs. Public-checkpoint and cross-platform tests show how the benefit varies across models and workloads. Training thus changes the acceptance–cost trade-off available to a deployment policy.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Block-Diffusion Drafting, Width Adaptation, Hardware-Aware Inference, Efficient LLM Serving

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_9bDaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The Authors present a block-diffusion-based speculative decoding approach and observe a controversial phenomenon: with a fixed number of tokens, a model with narrower layout improves acceptance rate of draft tokens per block at narrow inference width; while decreasing acceptance of draft tokens for a wider width model.

To address this issue, they propose a cost-driven adaptation. They claim, that optimal training width should be assessed with respect to the developers' cluster to identify parameters with optimal latency. Then, the model should be trained on that narrow layout. They claim, that this will lead to a training shape optimized for latency. Evaluation width, in turn, should be adapted to it. In cost-driven adaptation, authors provide the serving engine that calibrates and selects best static deployment width with respect to the adapted head, yielding modest throughput gains.

Critical strengths

None.

Critical weaknesses

Weakness 1: The presented manuscript lacks logical flow and details. There is no exact definition of D(), V(), and H() in Eq(1). Eq(2) and its description does not give an intuition why changing w affects the loss of first drafted tokens. The questions that authors are answering in Paragraph 3 are different from Paragraphs 4,5,6 titles, which makes the understanding of the manuscript extremely difficult to follow. The paper doesn't explicitly explain how to assess the training width with respect to the hardware. The main text of the manuscript does not have the explanation why width-10 is faster than width-9 on their hardware configuration and complimentary references to the appendix are also missing.

Weakness 2: the results show only marginal and inconsistent improvements. The speedup on Qwen3-8B are +1.58% on the primary benchmark and +0.69% on the secondary tests, both of which falls inside hardware noise threshold (3-5%, Line 1269). The manuscript misses architectural ablations and explanation of the results. The presented results on Qwen3-27B cannot be considered as an improvement being statistically negligible. Furthermore, the shown results on Llama are negative and only added to the appendix, therefore, this methodology, even when adapted to the hardware, is not universal and very specific to only one model inside one specific family.

These two collections of weaknesses are of major concern; and this manuscript needs substantial reformulation and revision to be considered for future acceptance.

Questions for authors
  1. Can you explain how to properly evaluate latency-wise optimal training width?
  2. Can you explain why this methodology does not transfer to a bigger Qwen model or a different model like Llama?
Reviewer_pDpsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work explores how the block size of the diffusion drafter based speculative decoding affects the acceptance length and decoding speedup. This work first shows that for the existing dFlash drafter trained with block size = 16, when the target model only checks the first 7 draft tokens, a width-8 drafting accepts 3.19% fewer tokens than a width-17 drafting whose remaining 9 draft tokens are simply discarded. This work believes that this is because diffusion drafters use full attention and the larger the block size used in the training, the better the diffusion drafter learns. Second, starting from this draft, this work continues training it for 500 updates at either width 10 or width 16 while keeping the supervised positions identical. The width-10 drafting improves acceptance at narrow inference widths and slightly lowers it at width-17 drafting.

Critical strengths

The research logic is clear in this work. The three training setups differ only in block width, and the results hold across three seeds with clear results. This work also honestly reports cases where their method does not help, such as on other models, larger batch sizes, and long contexts.

Critical weaknesses

(1) As the paper itself acknowledges, DFlash already evaluates crossed training and inference block sizes and reports asymmetric cross-width generalization. That the number of mask positions affects predictions at other positions follows naturally from bidirectional attention within a block, and that training closer to the deployment setting improves deployment performance is a well-known principle.

(2) To support the claim that “training reshapes width-dependent acceptance”, the drafters should be trained from scratch at different widths with matched data and compute. Since all heads are finally deployed at width 8, a drafter trained directly at width 8 should also be included in the main comparison.

Questions for authors

Please see weaknesses

1 comment

I do not see any response from author. I check the other reviews and find several of their concerns make sense. I think it is not very enough to be accepted. I highly do not recommend to accept this paper.

Reviewer_TCe8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how training width affects acceptance in block-diffusion speculative decoding. With supervision and verification capacity held fixed, narrower training improves acceptance at narrow deployment widths, yielding a 1.58% average throughput gain in the main serving experiment.

Critical strengths

The paper provides controlled experiments that separate training width from supervision length, offering some empirical insights into width-dependent acceptance behavior.

Critical weaknesses

The methodological novelty is very limited. The main approach essentially fine-tunes the drafter at a different block width and selects a deployment configuration through profiling. Similar training–inference width mismatch issues have already been studied in DFlash, and the proposed approach lacks a sufficiently novel algorithmic contribution.

The practical improvement is marginal. The main throughput gain is only 1.58%, while the acceptance improvement at the deployed width is approximately 0.049 tokens per block. Such limited gains are difficult to justify given the additional training and configuration selection costs.

The generalization of the findings is questionable. Most controlled experiments focus on Qwen3-8B, while results on other models and workloads are mixed. For example, Llama-3.1-8B shows negative throughput gains on CUDA, and the improvement on the 27B model is within the reported uncertainty. The benefits also appear sensitive to hardware configurations.

The writing and presentation need substantial improvement. The paper introduces numerous configuration variants and evaluation metrics without sufficiently clear explanations or a coherent narrative. The discussion frequently shifts between different experimental settings, making the central contributions difficult to follow. Overall, the presentation feels fragmented and insufficiently polished.

The main paper is overly short and lacks sufficient technical discussion. Important experimental details, implementation choices, and analyses are largely deferred to the appendix. The main text provides limited explanation of why the observed training-layout effects occur and insufficient discussion of the method's broader implications. This weakens the overall technical depth and clarity of the contribution.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.