- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents SPEAR, a method for self-speculative decoding in autoregressive order with diffusion language models. They show theoretically that the distribution generated by this algorithm matches the oracle distribution. They also analyze that the speedup comes from GPU parallelism, rather than a FLOP-advantage. They show that empirically, this method is superior to baselines in some cases, and does not provide much performance boost in other cases.
Critical strengths
- The method is theoretically sound.
- The batching of the "sequential" verification steps is a clever workaround to make sure that we can do verification in only one call.
- The problem is well-motivated in that the left-to-right order typically provides the best performance.
- The method leverages the parallelism of dLLMs without sacrificing quality.
- The authors provide an honest analysis of when this method has advantage, with respect to compute-boundedness on GPUs.
Critical weaknesses
- The gain of the method is marginal, and is only limited to cases where the GPU is not compute-bound. That being said, there are probably some practical cases where we can have full parallelism without compute-boundedness, so I don't think this is that much of a problem.
- In the presentation of the results, you should add a note that just because the speedup disappears in compute-bound scenarios doesn't mean that the method is useless: there are many practical examples where we aren't compute-bound (you should provide some examples to convince readers).
Questions for authors
- Why does AR order provide the best performance? Perhaps this was already shown in prior literature, but please put the answer in this paper. If you don't have an answer, at least note this for future work.
- Do you have results on other kinds of GPU hardware? Like A100s? It's ok if you don't, since I know GPU hardware could be hard to procure.
- You should clarify that you take autoregressive order to mean left-to-right. Technically, autoregression doesn't have to be left-to-right, and can be done in some arbitrary permutation (e.g., right-to-left).
2 comments
response to authors
Sg. Thanks for the clarifications, which have mostly addressed my concerns.
I think the paper is around a 3.5 now, and I would recommend acceptance. (In a previous year with a more fine-grained rating scale, I would have raised the score field, but the ratings this year are too coarse.)
Response to Reviewer Fvrn
Response to Reviewer Fvrn
Thank you for the careful and fair review. We followed each suggestion. Changes in the revised PDF are in blue, and the main addition is a new Section 4.4 (RQ4) with Appendices D.6 and D.7.
W1 / W2. The gain is marginal and limited to non-compute-bound settings; give examples where it is not compute-bound. We agree and now say this in the paper, with measurements. New Section 4.4 times LLaDA-8B forwards on rows of 32 to 1024 tokens at 1 to 16 rows per batch. It also times a 32-token row over a cached prefix on the 7B Qwen2 backbone of the block-causal Fast-dLLM v2. A cost model built from these timings reproduces our measured speedups at L = 4, 8 and 16 within 0.01 when given the measured tokens per block (within 0.02 with an i.i.d. acceptance model). Two findings frame the regimes.
Practical regimes where exactness is cheap, now listed in Section 4.4:
We also name the regime where it does not pay: when a batch already fills the GPU. Four samples of one prompt per pass remove the gain on LLaDA-8B and cut it to 1.45x on Fast-dLLM v2.
Q1. Why does AR order give the best performance? We added a paragraph to Section 2.
Q2. Results on other hardware such as A100? Our cluster has only H100 GPUs, so we could not measure an A100. We added a roofline estimate (Appendix D.6). An A100 does about three quarters of an H100 NVL's arithmetic per byte of weights (156 vs 214 FLOP/byte at peak), so rows become compute-bound sooner. Without per-call overhead, an 8-row batch of 32-token rows has alpha = 0.29 on an A100 against 0.21 on the H100, and 384-token rows have alpha = 1.00 on both. So we expect a smaller gain on bidirectional canvases and the block-causal regime to remain. We state that these are estimates.
Q3. Clarify that AR order means left to right. Done. The introduction now says that AR order means left to right in this paper. Section 2 adds that nothing depends on it. For any fixed permutation sigma, "next position in sigma" defines a policy, and SPEAR samples it exactly when each verification row reveals a sigma-prefix.
Thank you again. We hope the new section answers your point about practical regimes, and we would welcome further comments.