acceptodds
Under review as a conference paper at ICLR 2027

Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding

17 views

Abstract

Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier. We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with final conditional decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang. Across mathematics, coding, and conversation on Qwen3-4B and Qwen3-8B, DSpine raises the mean acceptance length of Qwen3-8B under greedy decoding from 3.77 with DFlash to 4.82 (+27.8%) and delivers 23.3% higher SGLang throughput than DFlash.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, block diffusion, token dependency, LLM inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hGYROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces DSpine, which is a parallel speculative decoding method based on depthwise causal conditioning. It establishes the token dependencies progressively across the depth dimension of the drafter. It has demonstrated the empirical observation clearly 1) token-predictive information appears in shallow layers 2) early predecessor conditioning improves successor prediction. Experiments are conducted fairly as all methods are trained from scratch using the same dataset and it shows significant improvement in both acceptance length and end-to-end inference speedup.

Critical strengths
  1. The paper is well motivated with clear empirical evidence on the how the depth and token dependency affect the final performance.
  2. The architecture for the drafter is well designed. The proposed layerwise feature injection, embedding supervision and final conditional decoding are sound.
  3. DSpine consistently outperforms strong parallel drafting baselines across different benchmarks under both greedy and sampling-based decoding and this gives pretty convincing results.
Critical weaknesses
  1. The ablation study section is quite short with brief mention of the results. I think the ablation study missing an important study on the effect of block-causal attention. DSpine changes the backbone attention from bidirectional to block-causal while simultaneously introducing adjacent causal injection. Both changes could improve predecessor-to-successor information propagation. However, the current ablations do not sufficiently isolate their individual effects.
  2. The gated adjacent injection resembles a lightweight residual feature communication mechanism. It remains unclear why the authors choose to project hidden states into a PCA-whitened output-embedding space.
  3. Does the
Questions for authors
  1. As DSpine is a depthwise technique, how will the acceptance-length and latency change when varying the number of injection layers?
Reviewer_yVAROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DSpine, a parallel drafter that introduces adjacent predecessor conditioning throughout the backbone. At each layer, a gated injection module transfers the predecessor’s predicted feature into the successor’s representation. A shared transfer space derived from the target model’s output embeddings, together with intermediate supervision, connects this process to final token-conditioned refinement. A transition cache precomputes adjacent candidate scores to reduce sequential decoding overhead. Experiments on Qwen3-4B and Qwen3-8B demonstrate improvements in acceptance length and SGLang serving throughput over several parallel drafting baselines.

Critical strengths
  1. The paper addresses an important problem with a well-motivated architectural design. Improving consistency among simultaneously drafted tokens is directly relevant to speculative decoding, where an early rejection invalidates the remaining draft. The proposed mechanism allows predecessor information to influence successor representations during backbone computation. The preliminary analyses provide useful motivation, particularly the timing experiments that hold the number of injection layers fixed while varying their placement.
  2. The method integrates representation learning and conditional decoding coherently. The shared output-embedding transfer space provides a common representation for predicted predecessor features and subsequently selected tokens. Intermediate supervision encourages these features to become informative early, while gated injection regulates their influence on successors. Reusing the same mechanism for final refinement makes the overall design conceptually appealing.
  3. The evaluation demonstrates practical inference benefits beyond acceptance-length improvements. The paper evaluates two target-model sizes, seven benchmarks, and both greedy and stochastic decoding for acceptance length. Serving experiments cover concurrency levels from 2 to 32 in SGLang, with absolute throughput results provided in the appendix. The reported average throughput improvements over DFlash2—11.1% on Qwen3-8B and 6.5% on Qwen3-4B—are encouraging, particularly because they are consistently observed across the evaluated serving configurations.
  4. The implementation and latency analysis support the practical feasibility of the approach. The transition cache and fused injection kernels address the additional computation introduced by conditional drafting. The paper reports approximately 33M additional parameters and provides a module-level latency breakdown, showing that the injection overhead does not offset the acceptance gains in the tested setting. These measurements strengthen the connection between the proposed architecture and its serving benefits.
Critical weaknesses
  1. Further ablations would help clarify the contribution of conditioning throughout depth. Table 2 provides useful evidence that both injection and final refinement improve acceptance length. Since final refinement reuses the last-layer injection mechanism, a comparison between all-layer injection and final-layer-only conditioning, while retaining the same refinement, would more directly establish the benefit of introducing predecessor information earlier. It would also be helpful to specify which architectural and training choices remain in the variant without injection and refinement, to clarify its difference from the DFlash baseline. Table 3 offers supporting evidence for intermediate supervision, although its contribution within the complete DSpine system remains less directly evaluated.
  2. The benefits of the specific design choices over DFlash2 could be isolated more clearly. The paper explains the architectural differences, including the supervised predictive-feature space and the shared mechanism for intermediate injection and final refinement. However, DFlash2 already incorporates dynamic adjacent-position mixing within the backbone and parallel scoring of adjacent candidate pairs. While the end-to-end improvements are encouraging, a controlled comparison of the two information-transfer mechanisms under matched attention, training, and final-decoding settings would help identify which aspects of DSpine account for the gains. This would strengthen the empirical justification for the proposed mechanism.
  3. A few additional implementation details would improve reproducibility. The paper specifies single-GPU execution, numerical precision, the attention implementation, and CUDA graphs, but I could not find the specific GPU model or SGLang version in the provided manuscript. Reporting these details, along with the number of timing runs and their variability, would make the serving results easier to reproduce and assess.
Questions for authors

Please check the weakness part above.

Reviewer_VhSoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DSpine is a parallel speculative drafter that injects predicted predecessor features throughout the backbone. It supervises these features in a space derived from the target's output embeddings and uses the selected predecessor token for final refinement. Candidate-pair scoring is batched, leaving a lightweight sequential selection step. Experiments cover Qwen3-4B and Qwen3-8B, seven benchmarks, and greedy and stochastic decoding. On Qwen3-8B under greedy decoding, mean acceptance length is 4.82 versus 4.39 for DFlash2. Greedy serving-throughput gains over DFlash2 average 11.1% on Qwen3-8B and 6.5% on Qwen3-4B.

Critical strengths

DSpine improves acceptance over strong retrained baselines, including DFlash2, on both target models and at both temperatures with common training settings. Under greedy decoding, it also improves throughput across the four serving workloads and tested concurrency levels. The diagnostic experiments support the design: shallow-layer readouts recover meaningful predictions, and providing accurate predecessor information earlier helps even with the number of interventions fixed. These findings, together with the measured serving gains, support conditioning through depth as a useful contribution.

Critical weaknesses
  1. The evaluation is limited to Qwen3-4B and Qwen3-8B. A direct comparison of acceptance length between DSpine and a well-trained DFlash2 baseline on Qwen3.8-27B and Qwen3.8-Flash-Next, under identical evaluation settings, would substantially strengthen the paper. The comparison should control prompts, decoding settings, and draft budgets, and document training budgets and convergence for both methods. This would establish whether DSpine’s gains extend beyond the evaluated model scales and remain meaningful against a strong, adequately trained baseline.

  2. Table 2’s nested ablations provide evidence for the contributions of last-write refinement and injection. However, the configuration removing both components does not explicitly specify whether block-causal attention and layer-wise supervision are retained. The authors should document these choices for every variant and hold them fixed when isolating injection. Otherwise, it remains unclear how much of the observed difference can be attributed specifically to the explicit injection pathway.

  3. Table 3 changes both the supervision objective and the readout parameterization: token CE uses the shared final RMSNorm and frozen LM head, whereas cosine supervision uses layer-specific learned projections. The comparison therefore does not isolate the effect of the supervision objective. A CE baseline with comparable layer-specific readout capacity would help distinguish the benefits of embedding supervision from those of readout flexibility. Without this control, the interpretation that intermediate token classification interferes with later contextual computation should remain tentative.

Questions for authors
  1. Can the authors compare acceptance length between DSpine and a well-trained DFlash2 baseline on Qwen3.8-27B and Qwen3.8-Flash-Next under matched evaluation settings to assess whether the gains extend to larger models?

  2. In Table 2, does the double-removal variant retain block-causal attention and layer-wise supervision? Given the nested removals and the separate timing interventions, how should we interpret the contribution of early injection alongside final refinement?

  3. Given Table 3's different readouts, how do you interpret the advantage of the embedding-supervision design and its relation to the contextual interference discussed in Section 5.5?

  4. Which GPU, workload, and concurrency were used for Figure 5's per-round comparison of DSpine and DFlash2, and how were the stage times measured and aggregated? How does this setup relate to the serving protocol in Appendix C?

1 comment

Thanks for your reply. Beyond just the sequence length, I would be more interested in seeing a fair comparison of throughput across different batch sizes for Qwen-3.8-27B, after all, there may be additional costs involved here. Furthermore, it is my understanding that the recommended block size for DFlash V2 on Qwen-3.8-27B is 8; using a block size of 16 seems quite unfair, as the model was likely not trained with a block size of 16.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.