- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SPOT, a speculative decoding method for diffusion language models. Its central observation is that stale target distributions often predict future committed tokens accurately, while predicting their commit order remains difficult. SPOT uses a 0.62M-parameter model to predict positions conditioned on hypothetical commitments and constructs a tree of candidate states. A shared-Gumbel verifier preserves the target sampling distribution under the stated decoding assumptions. Experiments report up to 6.8× fewer target evaluations and approximately 3.2× higher throughput, together with cross-model drafter transfer.
Critical strengths
-
A useful diagnosis of the drafting bottleneck. The position/token oracle experiments provide convincing evidence that commit-order prediction is particularly valuable for the studied decoding regime. The lightweight predictor directly addresses this bottleneck and improves over stale-confidence ordering. Using only probability features also makes the cross-model transfer results interesting.
-
A principled verifier with meaningful measured acceleration. The shared-Gumbel construction cleanly separates verification correctness from draft quality and supports tree-based speculation under sampling. The proof explicitly addresses adaptive position selection. Reporting both target evaluations and wall-clock throughput, including their different responses to draft budget, strengthens the practical evaluation.
Critical weaknesses
-
The comparisons do not fully establish the advantage over well-tuned competing methods. Appendix D.4 fixes the SSD-style chain depth at four, while SPOT can allocate substantially deeper paths. The baselines are adaptations within the authors’ decoder rather than complete original implementations. This is useful for isolating drafting, but limits broader superiority claims. Moreover, MRP is compared using published speedups measured with different hardware, block widths, implementations, and generation lengths. The assertion that these differences cannot explain the margin is insufficiently supported.
-
Inconsistent descriptions of the evaluation protocol weaken reproducibility. Section 5.1 states 350 samples, with all 164 HumanEval problems, whereas Appendix E describes first-100 evaluation subsets and explicitly associates Table 2 with those subsets. Appendix D.5 also justifies calibration-set separation by stating that every evaluated run uses samples 0–99. If the larger evaluation sets were used, that separation requires clarification. These inconsistencies affect the interpretation of sample coverage and hyperparameter selection.
Questions for authors
-
Protocol clarification: Which sample counts and exact splits produced Table 2? Were the MBPP samples used for mass-floor calibration excluded from every reported evaluation, including any 350-sample runs?
-
Comparison robustness: How do the conclusions change when baseline chain depth, graph structure, and verification budget are tuned for throughput using a comparable validation allowance? Can the MRP comparison be reproduced under matched conditions, or its superiority claim qualified?
-
Implementation exactness: Appendix A.1 assumes batch-invariant target evaluation and refers to Appendix G for finite-precision effects, but that appendix primarily discusses timing variability. What are the greedy trajectory mismatch rates between standalone and batched verification, especially for unpadded SPOT-γ? Similar downstream accuracy alone does not establish trajectory equality.
2 comments
Follow-up on our response
Dear Reviewer 6Tv8,
Thank you again for your time and effort.
We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.
We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.
Response to Reviewer 6Tv8
Dear Reviewer 6Tv8,
Thank you for the positive evaluation and for recommending acceptance. We especially appreciate your careful reading of the evaluation protocol and your questions on tuned baselines and a matched MRP comparison, which led to new experiments and clarifications that will be included in the revised paper and, we believe, improve it.
W1 / Q2. Tuned baselines and a matched MRP comparison.
Following the reviewer's suggestion, we strengthened the results with tuned baselines and added a matched MRP comparison, and tuning the baselines does not change the conclusions. Running the SSD-style chain as deep as the budget (8 and 20, first 100 problems of every task) changes its target passes by at most 4% and does not improve its speed, while SPOT still needs 16–47% fewer passes and is faster at every task and budget (table below). Spiffy uses its own offline-calibrated graph built with its published recipe, SPOT's own settings (the 0.6 runner-up threshold and SPOT-γ's mass floor) are fixed across all results, and with each method at its own best budget (8, 12 or 20), SPOT is 1.12–1.40× faster than the best baseline on every task at T=0 and T=1.
SSD-style chain at its main-table depth (4) and at depth equal to the budget, against SPOT. SDAR-8B, T=0, first 100 problems, NFE at budget 8 / 20.
For MRP we matched its own setting: block 16, where its released checkpoint exists, with the same prompts and the same verifier (SDAR-8B, T=0, SPOT at budget 8). SPOT still wins on every task, with 1.5–1.8× fewer target passes and 1.5–1.8× higher throughput, and running MRP's own released code with the same checkpoint confirms that our harness does not handicap it.
We thank the reviewer for these suggestions, which strengthen the comparison, and will add the tuned-baseline and MRP results, with both tables, to the revision.
W2 / Q1. Sample counts, splits, and the calibration samples.
We agree with the reviewer that the calibration point requires clarification and verification. The main table uses the sample counts and splits stated in Section 5.1. We checked γ under two settings: one in which the calibration problems were part of the evaluation set, and one in which they were excluded, using unused MBPP train problems. Both settings select the same value, γ=0.1, confirming that this does not affect the reported results. Nevertheless, we fully acknowledge the reviewer's important point and will revise the relevant sections accordingly to ensure complete consistency, clarity and reproducibility of the experimental protocol. We thank the reviewer for raising it and for helping us improve the paper's reproducibility.
Q3. Trajectory mismatch rates, especially for unpadded SPOT-γ.
Unpadded SPOT-γ matches a standalone run as often as padded SPOT does, and as often as draft-free decoding run at a different batch shape, so the differences come from the batch shape and not from drafting. Draft-free decoding at SPOT's own batch shape reproduces SPOT's output on 96% of problems. 94–99% of first differences are two nearly tied positions committed in swapped order. Correctness flips are balanced, so accuracy does not change. We thank the reviewer for this question and will clarify this in the revision where needed.
Thank you again. We hope the corrected protocol and the new checks resolve your concerns about soundness and comparisons. If they do, we would be grateful if you would take them into account in your final rating. We would be happy to answer any follow-up questions or clarify any remaining concerns.