acceptodds
Under review as a conference paper at ICLR 2027

SPEAR: Exact Self-Speculative Sampling from the Autoregressive-Order Policy of Diffusion Language Models

Abstract

Diffusion language models (dLLMs) can decode several tokens per forward pass and in any order, yet for reasoning the autoregressive (AR) order of the same model reaches a higher Pass@k. Sampling that AR-order policy exactly costs one forward pass per token, and the parallel decoders that avoid this cost sample a different distribution. We study whether exact sampling from the AR-order policy can be made cheaper on ordinary bidirectional dLLMs, and when it is worth its cost. We propose SPEAR, a training-free self-speculative sampler that drafts a block from one forward pass, verifies it against the exact AR-order conditionals in one batched pass, and keeps or resamples each token by the speculative sampling rule. We prove that its output follows the AR-order policy at any temperature. Experiments with three dLLMs on math and code show that the models agree with most of their own drafts, and that exact samples beat lossy parallel decoding at a matched number of forward passes. Matched on GPU time, the advantage disappears on math and survives on code only against a fixed-step decoder that breaks program syntax. Code is available at https://anonymous.4open.science/r/spear-code-iclr27-5C0B/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Masked diffusion language models, Self-speculative decoding, Exact sampling, Autoregressive-order generation, Batched prefix verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FvrnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SPEAR, a method for self-speculative decoding in autoregressive order with diffusion language models. They show theoretically that the distribution generated by this algorithm matches the oracle distribution. They also analyze that the speedup comes from GPU parallelism, rather than a FLOP-advantage. They show that empirically, this method is superior to baselines in some cases, and does not provide much performance boost in other cases.

Critical strengths
  • The method is theoretically sound.
  • The batching of the "sequential" verification steps is a clever workaround to make sure that we can do verification in only one call.
  • The problem is well-motivated in that the left-to-right order typically provides the best performance.
  • The method leverages the parallelism of dLLMs without sacrificing quality.
  • The authors provide an honest analysis of when this method has advantage, with respect to compute-boundedness on GPUs.
Critical weaknesses
  • The gain of the method is marginal, and is only limited to cases where the GPU is not compute-bound. That being said, there are probably some practical cases where we can have full parallelism without compute-boundedness, so I don't think this is that much of a problem.
  • In the presentation of the results, you should add a note that just because the speedup disappears in compute-bound scenarios doesn't mean that the method is useless: there are many practical examples where we aren't compute-bound (you should provide some examples to convince readers).
Questions for authors
  • Why does AR order provide the best performance? Perhaps this was already shown in prior literature, but please put the answer in this paper. If you don't have an answer, at least note this for future work.
  • Do you have results on other kinds of GPU hardware? Like A100s? It's ok if you don't, since I know GPU hardware could be hard to procure.
  • You should clarify that you take autoregressive order to mean left-to-right. Technically, autoregression doesn't have to be left-to-right, and can be done in some arbitrary permutation (e.g., right-to-left).
2 comments
AuthorsOpenReview ↗

Response to Reviewer Fvrn

Response to Reviewer Fvrn

Thank you for the careful and fair review. We followed each suggestion. Changes in the revised PDF are in blue, and the main addition is a new Section 4.4 (RQ4) with Appendices D.6 and D.7.

W1 / W2. The gain is marginal and limited to non-compute-bound settings; give examples where it is not compute-bound. We agree and now say this in the paper, with measurements. New Section 4.4 times LLaDA-8B forwards on rows of 32 to 1024 tokens at 1 to 16 rows per batch. It also times a 32-token row over a cached prefix on the 7B Qwen2 backbone of the block-causal Fast-dLLM v2. A cost model built from these timings reproduces our measured speedups at L = 4, 8 and 16 within 0.01 when given the measured tokens per block (within 0.02 with an i.i.d. acceptance model). Two findings frame the regimes.

  • The cost of a verification row decides the regime. Eight 32-token rows cost 0.12 of eight single rows (alpha = 0.12), and 0.21 for an engine with no per-call overhead. Eight 384-token rows, the size of a 256-token LLaDA canvas, cost 0.79 measured and about 1.00 without overhead.
  • So on a full bidirectional canvas, most of our measured gain is shared per-call overhead, and the paper says so. Any exact verifier that scores each prefix in its own row is capped at 1/alpha over stepwise decoding, whatever its acceptance (Section 3.2).

Practical regimes where exactness is cheap, now listed in Section 4.4:

  • Latency-bound decoding of one request at a time. On an H100, SPEAR measures 1.17 to 1.25x over stepwise there; interactive or on-device use is a likely fit that we did not measure.
  • Short verification rows, as in block-causal dLLMs (BD3-LM, Fast-dLLM v2, D2F, SDAR). They keep finished blocks in an exact causal cache, so a row is one block. SPEAR and its exactness proof apply unchanged. Measured on Fast-dLLM v2 7B (new App. D.7): exact SPEAR runs 2.61x faster than exact stepwise decoding. It is about as fast as our threshold baseline (the model's sub-block rule with temperature-one confidence) and more accurate, with Pass@1 +0.027 [+0.003, +0.051] on 200 GSM8K problems. The tempered-confidence variant is 1.25x faster than SPEAR, with accuracy unmeasured.
  • Outputs that lossy parallel commits break. On code, SPEAR beats CONF64 at equal GPU time (+0.10 to +0.14 Pass@k on HumanEval).

We also name the regime where it does not pay: when a batch already fills the GPU. Four samples of one prompt per pass remove the gain on LLaDA-8B and cut it to 1.45x on Fast-dLLM v2.

Q1. Why does AR order give the best performance? We added a paragraph to Section 2.

  • Ni et al. (2026) show the mechanism. Confidence-ordered decoding commits easy tokens first and postpones the high-entropy tokens where a solution branches, and Pass@k falls.
  • Our own control isolates order from step count (Appendix G.1). At an equal 256 passes, we detect no Pass@1 difference from order (+0.017, CI [-0.025, +0.061]), but order raises Pass@4 to Pass@16 by about 0.05, with intervals excluding zero.
  • A likely reason is that instruction data and chains of thought are written left to right, so each step conditions on the reasoning that justifies it. We label this as a hypothesis.
  • AR order is not best everywhere. On Sudoku, an adaptive order that fills easy cells first is far better (Kim et al., ICML 2025). We cite this and leave a task-level account to future work.

Q2. Results on other hardware such as A100? Our cluster has only H100 GPUs, so we could not measure an A100. We added a roofline estimate (Appendix D.6). An A100 does about three quarters of an H100 NVL's arithmetic per byte of weights (156 vs 214 FLOP/byte at peak), so rows become compute-bound sooner. Without per-call overhead, an 8-row batch of 32-token rows has alpha = 0.29 on an A100 against 0.21 on the H100, and 384-token rows have alpha = 1.00 on both. So we expect a smaller gain on bidirectional canvases and the block-causal regime to remain. We state that these are estimates.

Q3. Clarify that AR order means left to right. Done. The introduction now says that AR order means left to right in this paper. Section 2 adds that nothing depends on it. For any fixed permutation sigma, "next position in sigma" defines a policy, and SPEAR samples it exactly when each verification row reveals a sigma-prefix.

Thank you again. We hope the new section answers your point about practical regimes, and we would welcome further comments.

Reviewer_FvrnOpenReview ↗

response to authors

Sg. Thanks for the clarifications, which have mostly addressed my concerns.

I think the paper is around a 3.5 now, and I would recommend acceptance. (In a previous year with a more fine-grained rating scale, I would have raised the score field, but the ratings this year are too coarse.)

Reviewer_9hSTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+2)
Read review
Paper summary

The paper proposes SPEAR, a training-free self-speculative decoding method that samples exactly from the autoregressive (AR)-order policy of a bidirectional diffusion language model (dLLM). It uses the same model to draft multiple tokens in parallel, then verifies them using batched masked-prefix evaluations and the standard speculative sampling correction, which guarantees distributional exactness. While SPEAR reduces the number of forward calls by roughly 3.5× compared to stepwise AR decoding, it achieves only modest wall-clock speedups because bidirectional attention requires a separate verification computation for each prefix. Its acceleration primarily comes from amortizing model-weight memory transfers through batching, so the advantage disappears as GPU execution becomes compute-bound.

Critical strengths

This paper is clearly written, and exposes an important problem in self-speculative decoding for diffusion language models. In particular, it gives careful analysis of the speedup from SPEAR, and expose that in practical regime we are bounded by GPU memory bandwidth, and hence an anticipated speedup would disappear under current hardware. The conclusion is scientific and thought-provoking.

Critical weaknesses

The conclusion is drawn from a very narrow setting, where we do not train the model. This raises question on how much the intuition transfer to more practical scenario. For example what if we do not want an exact AR sampling, or we want to train a model through a self-speculative loss.

The biggest weakness is that despite clear analysis, it does not provide us with useful method to truly accelerate DLM decoding, nor does it establish that there is not much hope in this direction.

Questions for authors
  1. The paper spends some paragraphs talking about exactness of sampling. To my understanding it is just canonical speculative decoding. Is it mathematically trivial or am I missing something?
  2. Is there any practical regime SPEAR performs a lot better than AR models?
  3. What do the authors suggest to make the DLM inference system better based on the analysis of the paper?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9hST

Response to Reviewer 9hST

Thank you for the review. Your two main points led to the main addition: a ceiling for exact verification on bidirectional dLLMs and a measured account of where it lifts (new Section 4.4, Appendices D.6 and D.7). Changes are in blue.

W2. No useful acceleration method, and no proof that there is little hope. The revision now argues both sides with measurements.

  • The ceiling. Let alpha be the cost of an L-row batch relative to L single rows. A block commits at most L+1 tokens, so the speedup over stepwise is at most (L+1)/(1+alpha L) <= 1/alpha, strictly for alpha < 1. This holds for any exact verifier that scores each revealed prefix in its own row of a bidirectional canvas, at any acceptance rate (Section 3.2).
  • alpha, measured. On LLaDA-8B one row costs about 20 ms from 32 to 384 tokens, so the discount on a 256-token canvas (alpha = 0.79) is shared per-call overhead of an eager implementation. A roofline estimate without that overhead gives alpha = 1.00 there. Full bidirectional canvases therefore leave little room. At perfect acceptance and the best block length tested, our host-aware cost model gives at most 1.44x in our implementation and about 1.05x without per-call overhead.
  • Where it lifts. Short rows batch almost for free: alpha = 0.12 measured and 0.21 on the roofline for 32-token rows, both bidirectional and over a cached prefix on the Qwen2 backbone of Fast-dLLM v2. With measured tokens per block, our cost model reproduces the LLaDA speedups within 0.01 (0.02 with i.i.d. acceptance). Block-causal dLLMs have such rows: finished blocks sit in an exact cache, so a row is one block, and SPEAR applies unchanged. We ran it on Fast-dLLM v2 7B (App. D.7). After a fix that makes the batched forward bitwise batch-invariant (as on Dream-7B, it was not), exact SPEAR runs 2.61x faster than exact stepwise decoding (CI [2.55, 2.68]; the cost model predicts 2.79x). It is about as fast as our threshold baseline (the model's sub-block rule with temperature-one confidence) and more accurate: Pass@1 +0.027 [+0.003, +0.051] on 200 GSM8K problems. The tempered-confidence variant is 1.25x faster than SPEAR; we did not measure its accuracy. Against stepwise, the planned 4-sample test agrees on Pass@1 and answer distributions but one vote interval just excludes zero; an exploratory extension to 8 samples puts every interval around zero.

W1. Narrow setting: no training, exactness required.

  • Training for self-speculation. Training can raise acceptance. It cannot change alpha, which is set by the hardware and the row length. So the ceiling above holds for a fine-tuned model too.
  • Not requiring exactness. Then lossy decoders are the right tool, and our results already say so: at equal GPU time a threshold decoder matches or beats SPEAR on math (Table 4). The revision states this as a recommendation.

Q1. Is exactness just canonical speculative decoding? Yes. The proof of Theorem 1 is the standard speculative-sampling argument, and Section 3.3 now says we claim nothing new in it. Two things are specific to dLLMs. The proposal is the product of one-pass marginals, whose first position is already exact. And the theorem's condition, that a row returns the same logits inside a batch as alone, can fail. On Dream-7B it does (TV about 0.018 at batch 8), and batched verification then samples a different distribution with no visible error. We test the condition on each model.

Q2. Is there a practical regime where SPEAR is much better than AR decoding? Against stepwise AR-order decoding of the same model:

  • On LLaDA-8B: 3.5x fewer forward calls at the same distribution; 1.17 to 1.25x in GPU time for one request at a time, none with 4 samples per pass.
  • On the block-causal Fast-dLLM v2, a measured 2.6x (1.45x with 4 samples per pass). Against lossy decoders, the exact samples win per forward pass (Pass@1 +0.06 over CONF64) and on code at equal GPU time (+0.10 to +0.14 Pass@k over CONF64). If "AR models" means autoregressive LMs, we make no claim: they verify a draft in one causal pass, which a bidirectional dLLM cannot do.

Q3. What should DLM inference do, based on this analysis? Section 4.4 and the conclusion now give four recommendations.

  1. For test-time scaling on today's bidirectional dLLMs, use a lossy threshold decoder; exactness does not pay in GPU time.
  2. Use exact sampling when the distribution is the object, for example when estimating what the AR-order policy can solve, and on outputs that parallel commits break, such as code.
  3. Exact and fast needs cheap prefix-conditional scoring. Block-causal attention with exact caches provides it, and so do any-subset models (Guo and Ermon, 2026).
  4. Report GPU time next to forward calls. Any method that batches rows per call, including LoPA's branches, can look fast in calls and not in time.

We would value your view of the revised claims.

Reviewer_8tjyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPEAR, a training-free self-speculative decoding method for diffusion language models that enables exact sampling from their autoregressive-order policy. SPEAR uses the same model to generate draft tokens in parallel and verify them through speculative sampling, preserving the target distribution at any temperature. Experiments on mathematical reasoning and code generation demonstrate high draft acceptance rates and improved sampling quality under matched forward-pass budgets. However, the actual wall-clock speedup is limited due to the computational overhead of batched verification.

Critical strengths

The paper is written clearly and is easy to understand. The paper provides a very comprehensive and objective analysis of the limitations of the proposed method.

Critical weaknesses
  1. Although the proposed method significantly reduces the number of forward passes, the actual GPU runtime is barely reduced because the verification cost is too high, which makes me question the practical significance of the proposed method.
  2. The core idea of the paper is not novel, as similar methods have been proposed in WINO [1] and LOPA [2]. Therefore, the novelty is limited.
  3. The paper uses the potential value of exact on-policy sampling for RL as an important motivation, but it does not provide sufficient experiments to validate this advantage.
  4. The figures and tables in the paper are presented very poorly and have substantial room for improvement.

[1] Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs

[2] LoPA: Scaling dLLM Inference via Lookahead Parallel Decoding

Questions for authors
  1. What is the practical advantage of SPEAR?
  2. What are the key contributions of SPEAR, as the core ideas have been proposed in the existing method WINO [1] and LOPA [2]?
  3. Are there any evidence to support the potential value of exact on-policy sampling for RL of the proposed method?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 8tjy

Response to Reviewer 8tjy

Thank you for the review. Changes in the revised PDF are in blue. New: a WINO baseline (App. I.3), when exactness saves GPU time, incl. a block-causal dLLM (Sec. 4.4, App. D.6-D.7), an RL rollout-mismatch measurement (App. E), and revised figures and tables.

W2 / Q2. Novelty relative to WINO and LoPA. Both draft several tokens and then check them, as SPEAR does. The check differs, and it decides what is sampled.

  • WINO commits positions above a lenient confidence threshold and re-masks a token whose probability at a shadow position falls below a strict one. LoPA decodes several filling orders in parallel branches and keeps the most confident branch. Both compare a confidence with a threshold, are evaluated greedily, and target no distribution.
  • SPEAR keeps a draft with probability min(1, pi_AR/q) and resamples from the residual on rejection, so its output is exactly the AR-order policy at any temperature (Theorem 1). On the real models, paired Pass@1 intervals against stepwise contain zero in all six cells.
  • New WINO baseline (App. I.3; our port matches the official code bitwise). At tau = 0.6 on the 96 GSM8K problems of Table 2, WINO uses 39 calls and 1.27 s per sample (SPEAR 5.68 s). SPEAR leads it at Pass@8 by +0.064 [+0.023, +0.110] and in a 16-sample vote by +0.067 [+0.006, +0.127]. WINO is not exact: against stepwise its answer distribution has excess TV +0.109 (p < 0.001), the largest of all decoders; SPEAR's is -0.002 (p = 0.65). At equal GPU time the vote difference is -0.010, +0.027 and +0.055 at 10, 20 and 40 s, all intervals spanning zero. On HumanEval SPEAR leads WINO at Pass@8 by +0.149 [+0.083, +0.216].

We do not claim draft-and-verify. Our contributions are:

  • exact AR-order sampling on ordinary bidirectional dLLMs, with its practical condition (batch invariance, which fails on Dream-7B and Fast-dLLM v2 out of the box);
  • a cost analysis: exact verifiers of this kind are capped at 1/alpha over stepwise, where alpha is how cheaply the hardware batches verification rows;
  • evidence on what exactness buys under both budgets.

W1 / Q1. Practical advantage, given the small GPU-time gain. We agree the gain is small on bidirectional dLLMs. Sec. 4.4 explains why, and where that changes.

  • A 384-token LLaDA row costs about the same 20 ms as a 32-token row, so the discount there (alpha = 0.79) is per-call overhead; without it alpha is about 1.
  • Short rows batch almost for free (alpha = 0.12). Block-causal dLLMs keep finished blocks in an exact cache, so a row is one block. On Fast-dLLM v2 7B (App. D.7), exact SPEAR runs 2.61x faster than exact stepwise, about as fast as our temperature-one-confidence threshold baseline, and more accurate (Pass@1 +0.027 [+0.003, +0.051]). The tempered-confidence variant is 1.25x faster than SPEAR; its accuracy was not measured.

Practical advantages:

  • the block-causal result above;
  • 3.5x fewer calls than stepwise on LLaDA-8B (the other exact baseline evaluated here);
  • 1.17 to 1.25x over stepwise for one request at a time on LLaDA;
  • on code, where only 42 to 48% of CONF64's programs parse, a lead over CONF64 at equal GPU time (+0.10 to +0.14 Pass@k on HumanEval). For GPU-time-limited test-time scaling on bidirectional dLLMs, the paper now recommends a lossy threshold decoder.

W3 / Q3. RL as motivation without evidence. We agree. The motivation now rests on test-time sampling, the introduction mentions RL only as prior work, and we claim no RL improvement. App. E measures, on 32 GSM8K problems x 4 rollouts, how far each decoder's commit-time scores are from pi_AR, the policy GRPO differentiates. For lossy decoders these scores only proxy the behavior likelihood, because they omit which drawn tokens are kept and the sum over commit orders. The numbers are therefore diagnostics:

  • 17.4% (CONF64), 9.3% (WINO) and 6.4% (FASTDLLM) of tokens have a commit-score ratio outside the PPO clip range [0.8, 1.2], and every lossy rollout is outside it at the sequence level. For CONF64 the mean log-score gap per rollout is 111 nats.
  • A learner that treats these as pi_AR samples sees ratio 1 at its first step. The ESS of the proxy weights is 1.00 to 1.85 per group of 4.
  • For SPEAR the gap is zero by Theorem 1. Its verification rows reproduce the teacher-forced pi_AR log-probabilities bitwise, so GRPO gets them from the rollout pass. Whether the gap hurts learning needs multi-seed RL runs; our single-seed 600-step run detects no difference, and we say so.

W4. Figures and tables. Figure 2: fixed a y-axis that printed a spurious 10^-2, moved legends off the data, stated the sign convention. Table 1: grouped columns and removed check marks. Table 2: removed a bold on SPEAR's Pass@1 although FASTDLLM's is higher. Table 4: bold marks intervals that exclude zero. Captions say how to read each.

We would value your view of the revised paper.

Thanks for the author's response. I still concern about the limited contribution and novelty of the proposed method. So I will keep my score.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.