acceptodds
Withdrawn from ICLR 2027

One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive (AR) language models generate tokens sequentially, so generating long outputs is costly. Speculative decoding mitigates this by using a lightweight drafter to propose multiple tokens for the target AR model to verify in parallel, while a diffusion-based drafter can produce an entire draft block in one forward pass. In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width. Motivated by this efficiency bottleneck, we introduce Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration. Under a mild monotonicity assumption on the drafter, we further show that Packed-SS requires no more model calls than the standard two-call decoding in the greedy setting. Experiments on Nemotron 3B/8B and SetDLM 1.7B show higher throughput and lower mean wall time than both baselines under greedy decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion language models, Speculative decoding, Self-speculative decoding, Parallel decoding, Efficient language model inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C7VKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. To address this bottleneck, this paper introduces Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration.

Critical strengths
  1. The motivation in the paper is reasonable.

  2. The paper is clearly written.

Critical weaknesses
  1. The proposed method has too narrow an applicability, it can only be used with models like Nemotron-Labs-Diffusion that support both AR mode and diffusion mode.

  2. Although it reduces model calls, the final throughput improvement compared to Nemotron is limited.

  3. Is there a way to apply the proposed method to external drafters like dflash?

  4. The presentation of experimental results in the paper is unclear, it is strongly recommended to use tables to show the results.

  5. The figures in the paper have significant room for improvement.

Questions for authors

Please see the weaknesses

Reviewer_ydkJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Packed-SS, a self-speculative decoding schedule for models supporting both autoregressive verification and parallel diffusion drafting. It combines verification and drafting in one forward pass by conditioning the next proposal on an older verified prefix and retaining its unresolved suffix. This reduces per-call input size from quadratic to linear in speculative width compared with Quadratic-SS.

Critical strengths

The scheduling idea is simple and clearly presented. Separating proposal generation from the latest verification outcome enables a linear-size, single-call construction.

The left-to-right distribution-preservation argument appears sound, including under non-greedy sampling. The proof appropriately uses the original proposal distributions and addresses whether suffix acceptance preserves their conditional sampling.

Critical weaknesses

The practical gains are modest. At temperature 1, Packed-SS improves throughput over Linear-SS by only 2.0–6.3% and shows no consistent advantage over Quadratic-SS. The reduction from quadratic to linear input size therefore does not translate into a clear practical advantage in these settings.

The trade-off between draft quality and wasted computation is insufficiently analyzed. Advancing by A_t tokens leaves only W-A_t proposals for the next iteration. This creates a tension: higher acceptance, normally desirable in speculative decoding, also discards more of the concurrently generated proposal. A perfect drafter gives Packed-SS the same asymptotic tokens-per-call ceiling as Linear-SS. The paper does not establish which acceptance profiles make this trade-off beneficial.

The efficiency guarantee relies on an unvalidated assumption. Monotonicity requires that additional verified context never turns a correct draft prediction into an incorrect one. This does not follow from parallel drafting, and its empirical validity is not measured. Moreover, the theorem guarantees no more forward passes, rather than lower latency.

Questions for authors
  • How frequently is Assumption 3.1 violated in the evaluated models, and do those violations predict performance regressions? More broadly, when monotonicity fails or decoding uses a nonzero temperature, can the authors characterize when Packed-SS remains beneficial and when it incurs a slowdown?
  • How do retained draft length and subsequent acceptance vary with the preceding advancement? How does the speedup change as drafter quality improves?
  • Under which practical configurations does the linear-size construction provide a clear wall-clock advantage over Quadratic-SS beyond the reported temperature-1 results?
Reviewer_M4X7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Packed-SS, a self-speculative decoding method that combines autoregressive verification and diffusion drafting within a single forward pass. By generating new draft tokens based on the previous verified prefix and reusing the remaining suffix after verification, the method reduces the number of sequential model calls while maintaining linear complexity in the speculative width. The authors provide theoretical guarantees on distributional correctness and model-call efficiency under certain assumptions. Experiments on Nemotron and SetDLM demonstrate throughput improvements over Linear-SS and Quadratic-SS decoding baselines.

Critical strengths
  1. The paper proposes a simple and training-free decoding strategy that combines drafting and verification in a single forward pass with linear complexity.
  2. The paper provides theoretical guarantees on distributional correctness and model-call efficiency under certain assumptions.
  3. The experiments cover multiple model families and benchmarks, showing consistent throughput improvements over the selected baselines under greedy decoding.
Critical weaknesses
  1. Limited novelty. I-DLM [1] already supports simultaneous verification and drafting within a single forward pass with linear complexity. This challenges the claim in the abstract that "Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width." The paper does not sufficiently distinguish its contributions from this closely related work.

  2. Limited decoding efficiency. Given a speculative width W, Packed-SS has an asymptotic upper bound of (W+2)/2 on its average TPF (tokens per forward), since fully accepting the current draft leaves no draft tokens for the next iteration. In contrast, ISD [1] can theoretically achieve W+1 TPF under full acceptance. This suggests a structural limitation of Packed-SS in high-acceptance regimes, which is not sufficiently discussed or evaluated.

  3. Insufficient comparison with prior work. Despite the similarities with I-DLM [1], the paper does not provide a direct comparison in terms of acceptance rate, TPF, or end-to-end throughput, making it difficult to assess the practical advantages of the proposed method.

Questions for authors
  1. Could the authors clarify the key methodological differences between Packed-SS and I-DLM [1], particularly given that both support simultaneous drafting and verification with linear complexity?

  2. Could the authors provide a direct comparison with ISD [1] under comparable settings, including acceptance rates, tokens per forward pass, and end-to-end throughput?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.