- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The Authors present a block-diffusion-based speculative decoding approach and observe a controversial phenomenon: with a fixed number of tokens, a model with narrower layout improves acceptance rate of draft tokens per block at narrow inference width; while decreasing acceptance of draft tokens for a wider width model.
To address this issue, they propose a cost-driven adaptation. They claim, that optimal training width should be assessed with respect to the developers' cluster to identify parameters with optimal latency. Then, the model should be trained on that narrow layout. They claim, that this will lead to a training shape optimized for latency. Evaluation width, in turn, should be adapted to it. In cost-driven adaptation, authors provide the serving engine that calibrates and selects best static deployment width with respect to the adapted head, yielding modest throughput gains.
Critical strengths
None.
Critical weaknesses
Weakness 1: The presented manuscript lacks logical flow and details. There is no exact definition of D(), V(), and H() in Eq(1). Eq(2) and its description does not give an intuition why changing w affects the loss of first drafted tokens. The questions that authors are answering in Paragraph 3 are different from Paragraphs 4,5,6 titles, which makes the understanding of the manuscript extremely difficult to follow. The paper doesn't explicitly explain how to assess the training width with respect to the hardware. The main text of the manuscript does not have the explanation why width-10 is faster than width-9 on their hardware configuration and complimentary references to the appendix are also missing.
Weakness 2: the results show only marginal and inconsistent improvements. The speedup on Qwen3-8B are +1.58% on the primary benchmark and +0.69% on the secondary tests, both of which falls inside hardware noise threshold (3-5%, Line 1269). The manuscript misses architectural ablations and explanation of the results. The presented results on Qwen3-27B cannot be considered as an improvement being statistically negligible. Furthermore, the shown results on Llama are negative and only added to the appendix, therefore, this methodology, even when adapted to the hardware, is not universal and very specific to only one model inside one specific family.
These two collections of weaknesses are of major concern; and this manuscript needs substantial reformulation and revision to be considered for future acceptance.
Questions for authors
- Can you explain how to properly evaluate latency-wise optimal training width?
- Can you explain why this methodology does not transfer to a bigger Qwen model or a different model like Llama?
I do not see any response from author. I check the other reviews and find several of their concerns make sense. I think it is not very enough to be accepted. I highly do not recommend to accept this paper.