acceptodds
Under review as a conference paper at ICLR 2027

DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair

Abstract

Parallel drafting reduces the drafting overhead of speculative decoding for large language models (LLMs), but its gains remain limited by the accepted prefix length. Even when the correct token is present in the candidate pool, a single early selection error prevents subsequent predictions from being used. We propose DRelay, which uses global information from the entire draft block to perform prefix-aware selective repair of candidate selections before target-model verification. DRelay bases its decisions on both candidate correlations and the selected path: a global reader extracts predictive information across positions for each candidate. While a causal selector combines candidate-level information extracted by the global read with the tokens selected at preceding positions to determine whether the native choice at the current position is consistent with the global evidence and the selected prefix. It then decides whether to retain or replace the token, thereby repairing early errors and extending the accepted prefix. We further jointly train the draft backbone and the selector, combining candidate-support learning with a repair objective, while weighting the repair loss according to each block position’s potential contribution to the consecutive accepted prefix. Across eight diverse benchmarks on an H800 GPU,DRelay consistently improves both average acceptance length and end-to-end decoding performance over DFlash, Domino, and DSpark. Under SGLang serving, DRelay improves average end-to-end speedup over DFlash, Domino, and DSpark by 14.7%–16.8%, 8.7%–9.3%, and 8.1%–9.3%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Diffusion Language Models, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s1MzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

DRelay addresses a limitation of speculative decoding: an early draft error can prevent later correct tokens from being accepted. It uses a Global Read module to gather candidate-specific information across the draft block, then a Causal Selector to keep or replace each token based on that information and the already selected prefix. The draft model and selector are trained jointly, with more weight on decisions that can extend the accepted prefix.

Critical strengths

S1. Their motivation is interesting and compelling: in SD's drafting, even if suffix draft tokens are correct, early token's rejection forces them all to be rejected. By correcting one or a few early rejection may increase the number of accepted tokens by much, which is interesting (Section 3.3).

S2. In the evaluation they included accuracy ablation across the loss proposals, which is interesting.

Critical weaknesses

W1. There is insufficient evidence that early errors can be corrected with the suffix's information. In other words, the meaning of Section 3.2 and Figure 2 is unclear. The reported log probability is calculated using the target model's information, making the experiment meaningless: it does not rule out at all that the accuracy improvement comes from the target model rather than from information added through the suffix. It should be changed to use the draft, and the experimental method should also be changed. For example, the authors should also test whether the probability decreases when the same method is applied to samples where the suffix is incorrect.

W2. The paper does not explain theoretically or experimentally why the proposed Global Read/Causal Selector approach can reduce early-token errors compared with existing methods that predict each token. Sections 4.2, 4.3, and 5 mainly describe how the proposed method works, but do not explain theoretically why it helps reduce early-token errors. Even the method itself is difficult to understand from the paper's descriptions.

Regarding this, I suggest experiments that seem necessary but are missing. The paper should separately report the frequency of REPAIR and an extended confusion matrix of selector decisions, with both probabilities and raw counts (e.g., 10/15), for the following cases: (1) correct native token → KEEP, (2) correct native token → REPAIR, (3) wrong native token → KEEP, (4) wrong native token → REPAIR with the correct token, and (5) wrong native token → REPAIR with another wrong token. Correctness should be defined against the target greedy reference. The paper should also report the average number of correct suffix tokens accepted when a correction is successful.

W3. The presentation needs significant improvements. The logical progression is unclear (the paper uses many unnecessary qualitative and figurative sentences and terms, and does not sufficiently connect its claims—for example, line 269: "A single summary can obscure competing continuations."). In addition, terms newly used in the paper (e.g., "addressable prefix memory," "candidate pool") are used without definition, and the variables in the equations (e.g., i, j, k, l, and g in Sections 4.2 and 4.3) are not sufficiently explained.

W4. The paper applies several heuristics (e.g., setting k to 16 when selecting top-k candidates in Global Read, and considering the past 64 tokens in the Causal Selector). It needs at least a brief justification for how these values were chosen.

Questions for authors

W1--W4 and following.

Q1. How much of the performance improvement comes from retraining the backbone rather than from repair?

Q2. If the correct answer is in the top-16 candidate pool in about 94% of cases, a simpler alternative would be to have the target directly verify the pool candidates using tree verification (e.g., EAGLE-style methods or TAPS), rather than having the selector choose a single path. What advantages does DRelay offer over this approach?

7 comments
AuthorsOpenReview ↗

Response to Reviewer s1Mz

We thank the reviewer for their comments. We are willing to consolidate the notation and clarify the experimental descriptions. However, we disagree with treating several misinterpretations of our offline diagnostics, methodological mechanisms, and tree verification as fundamental soundness concerns.

W1: Whether Figure 2 Demonstrates That Suffix Information Can Repair Early Errors

The purpose of Figure 2 is not to simulate the selector at deployment, but to investigate a more specific question: Does later draft content contain discriminative signals that can distinguish between earlier candidates?

The real suffix and control suffix use the same frozen target model, identical prefixes, current-position candidates, suffix lengths, and correctness conditions. The only difference is whether the suffix is aligned with the original continuation. Moreover, the target probability of the current-position candidate is explicitly excluded from the score. Therefore, the target model serves only as a common offline assessor in both groups and cannot, by itself, explain the observed difference.

Replacing the score with draft-model probabilities is not a valid correction to this experiment. Draft-model probabilities measure the draft model's internal consistency, whereas Figure 2 measures the compatibility between candidate tokens and the target continuation. These are fundamentally different questions.

With real suffixes, the correct candidate is selected in 46 out of 86 DFlash cases and 31 out of 69 Domino cases, representing improvements of 53.5 and 44.4 percentage points over the matched controls, respectively.

At inference time, DRelay does not access these target-model scores, future target tokens, first-error locations, or oracle filtering conditions. Figure 2 only establishes that later draft candidates contain exploitable cross-position signals. The practical benefit at deployment is independently validated by the Full versus No Global Read ablation.

Therefore, the reviewer's conclusion that this experiment is "meaningless" is unjustifiably strong. Deliberately constructing incorrect suffixes could serve as an additional negative control, but it is not a prerequisite for the validity of the existing diagnostic.

AuthorsOpenReview ↗

Response to Reviewer s1Mz

W2: Why Global Read and the Causal Selector Can Reduce Early-Token Errors

The paper already provides three levels of evidence.

First, the correct token remains in the top-16 candidate pool at the first error position in 94.0% of DFlash cases and 95.1% of Domino cases, and in 88.8% of Domino's actual served traces. This indicates that many failures arise from candidate selection rather than candidate generation.

Second, when the correct candidate is available and the subsequent draft continuation is correct, the continuation provides discriminative information for identifying the earlier correct candidate (46/86 for DFlash and 31/69 for Domino).

Third, the one-edit oracle increases accepted length by 1.65 and 1.22 tokens for DFlash and Domino, respectively, demonstrating that correcting a single early position can recover a subsequent run of correctly predicted tokens.

The two components of Global Read also serve distinct purposes.

Slot mixing first compresses the candidate distribution at each position into a summary and then mixes these summaries across positions to construct shared context for the entire block. However, candidates C and D at the same position share this summary, which alone cannot determine whether C or D should be selected.

Candidate reads then construct separate representations for individual candidates. Each candidate uses its own query to attend to candidates across the entire block, allowing C and D to retrieve different continuation evidence.

The Causal Selector subsequently makes causal decisions using a strictly maintained selected-prefix memory. If C is repaired to D at the current position, the next position conditions on D rather than the original C.

In other words, Global Read addresses how earlier candidates can exploit later information, while the Causal Selector addresses how a repair is propagated to subsequent positions.

The Full versus No Global Read ablation shows that accepted length decreases from 7.58 to 7.16 when Global Read is removed, directly validating the contribution of this information pathway.

Reducing DRelay to "independently predicting one token at each position" is inconsistent with both the equations and the implementation presented in the paper.

Regarding the reviewer's requested five-category KEEP/REPAIR statistics, we disagree that they are necessary evidence for establishing the validity of the method.

These statistics are post-hoc classifications of actions already taken by the selector. The overall REPAIR frequency can be obtained by summing the three corresponding categories, and raw-count examples such as "10/15" do not themselves provide additional mechanistic evidence. The correctness reference is already clearly defined as the target-greedy token under the same target prefix.

Likewise, the average number of accepted suffix tokens following a successful repair is a conditional decomposition of accepted length and cannot replace full-system metrics.

If included, these statistics should serve as supplementary diagnostics rather than as prerequisites for validating Global Read or grounds for dismissing the existing evidence.

W3: Terminology, Variables, and Readability

Both "candidate pool" and "addressable prefix memory" are standard concepts in speculative decoding, and the paper provides concrete definitions specific to our method.

Section 4.1 defines the candidate pool as the top-16 candidates retained at each position.

Section 4.3 defines the prefix memory as an addressable storage structure containing the anchor and previously selected entries, with a strict (j<i) attention mask.

The variables (i), (j), (k), (l), and (g) denote the current position, a previous position, the candidate index, the attended candidate node, and the gate logit, respectively.

Therefore, the reviewer's claim that these concepts are undefined is factually inaccurate.

At most, the definitions are distributed across two subsections, which constitutes a presentation issue rather than an undefined algorithm.

We will consolidate the notation and provide a clearer four-step description of the execution process. These improvements do not require any modification to the proposed method.

W4: The Choice of K=16 and the 64-Token History Window

K=16 represents an operating point balancing candidate coverage against selector overhead. The reported coverage results already demonstrate that this candidate pool retains the vast majority of potentially repairable tokens.

Increasing K incurs additional candidate-reading computation and memory overhead, whereas decreasing K risks excluding correct candidates.

The 64-token history window provides a bounded read over strictly verified history. Its size affects computational cost and performance, but it is not a prerequisite for the correctness of the proposed method.

Neither parameter is claimed to be a theoretically optimal constant across all models or workloads.

AuthorsOpenReview ↗

Response to Reviewer s1Mz

Q1: Does the Performance Improvement Come from Backbone Retraining or Repair?

The "No selector loss" configuration retains the draft objective while disabling repair, achieving an accepted length of 7.22, compared with 7.58 for the full model.

This already demonstrates an additional improvement of 0.36 tokens associated with the complete repair-enabled design. Therefore, the entire performance improvement cannot simply be attributed to backbone retraining.

A more detailed decomposition of frozen-backbone versus jointly trained-backbone effects would require an additional 2×2 factorial experiment.

However, this is a different question from whether repair provides additional benefits. The existing ablation already addresses the latter.

Q2: Why Not Directly Verify the Top-16 Candidate Pool Using Tree Verification?

The reviewer conflates candidate coverage at a single position with coverage of a complete target-conditioned verification tree.

The reported 94.0% only indicates that the correct token appears in the top-16 marginal candidate pool at a particular error position. It does not imply that subsequent positions already have target-model states conditioned on that token, nor does it guarantee that a complete correct path exists in the verification tree.

Verifying multiple branches requires the target model to compute nodes under different prefixes and maintain the corresponding KV-cache states. This changes both target-side computation and memory requirements.

In contrast, DRelay first uses draft-side candidate information to select a single path and then performs conventional single-path target verification.

Therefore, tree verification is not an equivalent or inherently simpler replacement for DRelay.

A fair comparison must match the target checkpoint, hardware, inference backend, number of target-scored nodes or FLOPs, and end-to-end latency.

Our claims are explicitly limited to single-path parallel speculative decoding. We do not claim that DRelay outperforms tree-based verification under every possible verification budget.

Reviewer_4ox2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets the waste caused by first-error truncation in parallel drafting. A diagnostic study shows that at the first rejected position the correct token is still among the drafter's top-16 candidates in 88.8%-95.1% of cases, that information from later block positions helps discriminate among candidates, and that an oracle single repair would add 1.22-1.65 committed tokens. The proposed DRelay combines a Global Read, which applies candidate-specific attention over the block's top-K candidates across positions, with a Causal Selector that makes KEEP/REPAIR decisions conditioned on an addressable memory of the already selected prefix and the verified target history.

On Qwen3-4B and Qwen3-8B, across eight math, code, and chat benchmarks at T=0 and T=1 on a single H800 with SGLang, DRelay reports average end-to-end speedups of 2.81x-3.89x, including its 0.55 ms selector overhead, which is roughly 8%-10% above the strongest of DFlash, Domino, and DSpark in each setting, and it remains fastest at concurrency 2-32 on AIME25 and HumanEval.

Critical strengths

The top-16 coverage analysis distinguishes candidate-selection failures from candidate-generation failures, the continuation analysis shows that later positions carry evidence about earlier choices, and the oracle repair experiment quantifies the available headroom. The paper also explicitly separates these oracle opportunity estimates from deployed inference rather than claiming that oracle gains are realizable.

The selector raises per-round latency from 6.38 ms to 6.93 ms (8.6%), yet DRelay has the highest average speedup in all four model/temperature panels of Table 1 and at every tested concurrency level, with speedups that include prefill and all decoding overhead. Consistent improvements over three strong parallel-drafting baselines in measured serving conditions make the contribution practically relevant.

Critical weaknesses

The stochastic decoding procedure is not specified. Section 4.3 describes verification with an ordinary greedy rule, but Table 1 also reports T=1 results. Because the deterministic KEEP/REPAIR step changes the proposed tokens, it is unclear which proposal probabilities enter the acceptance test and the residual distribution, and therefore whether the T=1 procedure preserves the target distribution, which is central to standard speculative sampling.

Attribution of the gains to specific components is incomplete. Removing Global Read lowers mean accepted length from 7.58 to 7.16, but this is a single setting without seed variation, timing, or a capacity-matched local selector, so it does not isolate the contribution of cross-position candidate information. Prefix memory, verified-history access, and the gate/rank decomposition are not separately ablated.

Questions for authors
  1. What exact proposal, acceptance, and correction procedure produces the T=1 results, which probability is used for a repaired token in the acceptance ratio and residual distribution, and why does the procedure preserve the target distribution?
  2. How does Global Read compare with a parameter-matched selector restricted to local candidate information, under matched training and measured end-to-end latency, and do the main gains persist across training seeds?
1 comment

Thank you for the positive assessment and constructive comments. Omitting the T=1T=1 rule in Sec. 4.3 was an oversight; for Global Read attribution, we add stage-isolating ablations and analyze capacity, latency, and stability.

W1 / Q1: Proposal, Acceptance, and Residual Correction at T=1T=1

Direct answer. KEEP/REPAIR is deterministic at all temperatures: the chosen path x1,…,xHx_1, \dots, x_H defines a point-mass proposal qi(v)=I[v=xi]q_i(v) = \mathbb{I}[v = x_i] at slot ii. Whether slot ii is KEEP or REPAIR, the proposal probability in both the acceptance ratio and residual distribution is qi(xi)=1q_i(x_i) = 1 (00 for v≠xiv \neq x_i); neither raw draft probabilities nor piactp_i^{\mathrm{act}} (Eq. 2) are used. Output sequences match target-only T=1T=1 sampling identically.

Per-round procedure. Given committed prefix cc:

  1. Proposal: Causal Selector sets x1,…,xHx_1, \dots, x_H left-to-right via gi+log⁡rik∗>0g_i + \log r_{ik^*} > 0 (no sampling, invariant to TT), defining qi=δxiq_i = \delta_{x_i}.
  2. Target forward: One pass on (c,x1,…,xH)(c, x_1, \dots, x_H) gives pi(⋅)=softmax(zitgt/T)p_i(\cdot) = \mathrm{softmax}(z_i^{\mathrm{tgt}} / T) at T=1T=1 (i=1,…,H+1i = 1, \dots, H+1).
  3. Acceptance and residual (Leviathan et al., 2023; Chen et al., 2023): With qi=δxiq_i = \delta_{x_i}, acceptance at slot ii is min⁡(1,pi(xi)/qi(xi))=pi(xi)\min(1, p_i(x_i)/q_i(x_i)) = p_i(x_i), and on rejection [pi−qi]+[p_i - q_i]_+ normalizes to pi(v)/(1−pi(xi))p_i(v) / (1 - p_i(x_i)) (v≠xiv \neq x_i). Checking i=1,…,Hi = 1, \dots, H sequentially, at the first rejection i=A+1i = A+1 we sample yA+1y_{A+1} from this residual and discard later proposals (if all HH pass, sample yH+1∼pH+1y_{H+1} \sim p_{H+1}), committing x1,…,xA,yA+1x_1, \dots, x_A, y_{A+1} (A+1A+1 tokens, matching τ\tau in Sec. 6.1). As T→0T \to 0, this reduces to the greedy rule.

Why the target distribution is preserved. Given (c,x1,…,xi−1)(c, x_1, \dots, x_{i-1}), slot ii emits xix_i with probability 1⋅pi(xi)=pi(xi)1 \cdot p_i(x_i) = p_i(x_i) and any v≠xiv \neq x_i with (1−pi(xi))⋅pi(v)/(1−pi(xi))=pi(v)(1 - p_i(x_i)) \cdot p_i(v)/(1 - p_i(x_i)) = p_i(v)—identical to pi(v)p_i(v) for all vv (equivalent to drawing yi∼piy_i \sim p_i and taking the longest matching prefix). This holds regardless of KEEP/REPAIR and needs no calibration; the selector affects only E[A]\mathbb{E}[A]. Bias arises only if tokens are deterministically rewritten while pre-repair draft probabilities enter the acceptance ratio, which we do not do.

Revision. Unify T=0T=0 and T>0T>0 verification in Sec. 4.3, and add the T>0T>0 algorithm and proof to the appendix.

W2 / Q2: Global Read Attribution, Capacity, Latency, and Training Seeds

Two-stage ablation. Under Table 3's setup (Qwen3-4B, T=0T=0, H=15H=15, K=16K=16, 8-benchmark mean), we trained two variants from scratch, each removing one Global Read stage:

Variantτ\tau (8-benchmark mean)
Full (slot mixing + candidate reads)7.58
Slot mixing only (w/o candidate reads)7.32
Candidate reads only (w/o slot mixing)7.21
No Global Read (Table 3)7.16

Over No Global Read (7.16), slot mixing alone adds +0.16 and candidate reads alone adds +0.05, while both together add +0.42 (twice their sum), showing synergy between 15-slot mixing and 240-node candidate reads.

Capacity and local selector. No Global Read in Table 3 is a matched-training local selector retaining all of Causal Selector (projections, 64-token history, prefix memory, shared residual MLP, Gate/Rank): 32.81M parameters (93.6% of Full's 35.06M) vs. 2.25M (6.4%) for Global Read. Against Table 4, going from No selector loss (τ=7.22\tau = 7.22, repair disabled, Sec. 6.3) to the 32.81M local No Global Read (τ=7.16\tau = 7.16) brings no gain from >32>32M local parameters (without future block evidence, local edits can corrupt correct native tokens), whereas adding 2.25M in Global Read raises τ\tau to 7.58 (+0.42). Gains thus stem from cross-position candidate evidence rather than capacity (we will note the 6.4% gap).

Latency. While variants were not re-timed end-to-end, Fig. 4 bounds the cost: the 35.06M selector takes 0.55 ms/round (6.38 →\to 6.93 ms), included in Table 1's 3.57× speedup. Since Full raises τ\tau by 5.9% over No Global Read (7.58 vs. 7.16), Full is faster end-to-end whenever the 93.6%-parameter, 15-step Causal Selector takes ≥0.17\ge 0.17 ms (i.e., parallel Global Read takes ≤0.38\le 0.38 ms).

Training seeds. Each variant trains from scratch on 1.42M PerfectBlend across eight H800 GPUs; like DFlash, Domino, DSpark, and LiLiCorr, we report single-seed runs and will note this. Stability is supported by: (1) three independent trainings on Qwen3-4B, 8B, and 3.5-4B giving consistent τ\tau gains over DFlash (+1.43, +1.64, +1.41 at T=0T=0); (2) higher τ\tau in all 32 cells of Table 1 (vs. DFlash, Domino, DSpark), +0.52 to +0.62 above the strongest baseline in panel means; and (3) a monotonic progression (7.16 →\to 7.21 →\to 7.32 →\to 7.58).

Reviewer_tY2mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRelay, a method for speculative decoding that uses a parallel draft block as both a candidate source and a context for candidate selection. The core idea is to perform a global read over the draft block, then use a causal selector to decide, at each position, whether to keep the original draft token or repair it with an alternative from the top-K candidate pool. The repaired path is then submitted to the target model for verification. Experiments on Qwen3-4B and Qwen3-8B across eight benchmarks show consistent improvements over DFlash, Domino, and DSpark, with up to 4.49× end-to-end speedup over autoregressive decoding.

Critical strengths
  1. Clear problem formulation and presentation. The paper identifies a practical and important problem in parallel speculative decoding: early errors truncate the accepted prefix even when later predictions are correct. The writing is clear, and the diagnostic analysis in Section 3 effectively motivates the proposed repair mechanism.
  2. The proposed method is technically sound and well-structured. The separation into Global Read and Causal Selector is logical, and the training objective with prefix-value weighting is a reasonable design choice.
  3. Consistent empirical gains. DRelay outperforms the compared baselines on all eight benchmarks and at both temperatures, demonstrating the effectiveness of the approach under the evaluated settings.
Critical weaknesses
  1. Missing comparison with DFlash2 [1] and LiLiCorr [2]. The authors’ first contribution is to select tokens from the candidate pool using information from the entire draft block. However, this idea has already been explored by two closely related works, DFlash2 and LiLiCorr, neither of which is compared against. Without these comparisons, it is impossible to assess whether DRelay’s additional complexity and overhead (0.55 ms selector cost, +8.6% total latency over DFlash) are justified, and the paper’s central claim of superiority over existing selection-based methods is not adequately supported.
  2. Design rationale for Global Read is insufficiently explained. The paper does not explain why Global Read requires both top-16 slot mixing and candidate reads, nor what motivates this particular two-stage design.
  3. Selector training cost is not reported. The paper states that the parallel drafting backbone, Global Read, and Causal Selector are trained jointly, but provides no details on the selector’s parameter count, training time, memory footprint, or convergence behavior. Given that the selector introduces non-trivial architectural components (bidirectional attention-FFN blocks, gating, ranking branch), the training overhead relative to lightweight alternatives (e.g., DFlash2’s selector, LiLiCorr’s head) should be quantified. Without this, it is difficult to evaluate the practical cost-benefit trade-off of the proposed design.
  4. Ablation study lacks experimental setup and cross-dataset analysis. The ablation in Table 3 is not accompanied by sufficient experimental details. It would be useful to conduct ablations on multiple datasets to analyze how the gains vary across different scenarios.
Questions for authors
  1. Can the authors provide a comparison with DFlash2 [1] and LiLiCorr [2] under identical target models, hardware, decoding budget, and evaluation protocol? Specifically, how does DRelay’s acceptance length, per-round latency, and end-to-end speedup compare to DFlash2’s and LiLiCorr’s?
  2. What is the design rationale for top-16 slot mixing and candidate reads? Can the authors provide ablations isolating the contributions of these two components?
  3. What is the parameter count and training cost (GPU hours, memory) of the Global Read and Causal Selector modules? How do these compare to DFlash2’s ~2M parameters/ ~0.6% latency and LiLiCorr’s ~36.76M parameters/~2.8% per-block latency?
  4. Can the authors provide experiments on linear-attention (or hybrid-attention) and MoE models? The current evaluation is limited to Qwen3 dense models, so the effectiveness of Global Read and Causal Selector on other architectures remains unclear.
  5. The ablation study in Table 3 lacks experimental setup details. Can the authors conduct ablations on multiple datasets to analyze the benefits under different scenarios?

References

[1] Inco AI. DFlash 2: Keep Drafting Parallel. Technical blog, August 2026. https://inco.ai/blog/dflash2/
[2] Rusanovsky, M., Miron, Y., Uziel, R., Belhasin, O., Zilberstein, R., Ashkenazi, M., & Elad, M. LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding. arXiv:2608.20530, 2026.

3 comments

Thank you for the thorough and constructive review. Below we address W1–W4 and Q1–Q5 point by point. We have added three sets of experiments: (1) a controlled comparison with concurrent works DFlash2 and LiLiCorr under identical settings; (2) an ablation isolating the two stages of Global Read; and (3) an evaluation on the hybrid-attention target model Qwen3.5-4B. We also provide a submodule parameter breakdown, inference/training cost and convergence analyses, and full ablation setup details with a cross-scenario analysis.

W1 / Q1: Comparison with DFlash2 and LiLiCorr

DFlash2's technical blog was released on August 18, 2026, and LiLiCorr (arXiv:2608.20530) appeared on August 20, 2026, with source code released in September. Both are concurrent with our submission and were therefore not included in the initial manuscript. We have now evaluated both under the exact protocol of Table 1 and will include them in Related Work and our experimental comparisons.

Setup. Target model Qwen3-4B, T=0T=0. We retrained DFlash2 and LiLiCorr on the same training corpus as DRelay (1.42M PerfectBlend) using their official model architectures and configurations. Inference uses one parallel draft pass plus one single-path verification per round (H=15H=15), SGLang on a single H800 GPU, and the same eight benchmarks as Table 1. τ\tau is committed tokens per round, and SS is end-to-end speedup over autoregressive decoding.

MethodSelector / Head Params (M)τ\tauSelector / Head Latency (ms)Total Round Latency (ms)SS
DFlash—6.15—6.383.11×
DFlash21.976.800.236.623.39×
LiLiCorr32.046.810.276.653.38×
DRelay35.067.580.556.933.57×

Per-benchmark accepted length τ\tau and end-to-end speedup SS:

MetricMethodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAlpacaAvg.
τ\tauDFlash6.407.897.326.616.017.404.113.486.15
τ\tauDFlash27.278.937.917.076.797.934.683.856.80
τ\tauLiLiCorr7.618.978.117.026.637.864.543.776.81
τ\tauDRelay9.979.628.377.627.098.235.274.497.58
SSDFlash3.05×3.99×3.60×3.24×3.47×3.76×1.96×1.83×3.11×
SSDFlash23.25×4.38×3.82×3.55×3.87×4.12×2.21×1.93×3.39×
SSLiLiCorr3.39×4.38×3.90×3.51×3.76×4.07×2.13×1.88×3.38×
SSDRelay4.18×4.49×3.85×3.63×3.84×4.07×2.35×2.15×3.57×

DRelay achieves higher τ\tau than DFlash2 and LiLiCorr on all eight benchmarks, improving the mean by +0.78 and +0.77 (+11.5% and +11.3%). Although total round latency is 0.31 ms and 0.28 ms higher (+4.7% and +4.2%), average end-to-end speedup remains 5%–6% higher (3.57× vs. 3.39× and 3.38×), demonstrating a clear end-to-end speed advantage.

Key differences. DRelay differs in how it combines block-wide future evidence with the full selected prefix while controlling false repairs:

  1. Decoupling global candidate reads from full-prefix causal decisions. Global Read first runs in parallel so each candidate at slot ii queries all H×K=240H \times K = 240 candidate nodes with its own query (Eq. 1); Causal Selector then decides left-to-right conditioned not only on adjacent transitions, but on all earlier chosen entries Pi=[x0,…,xi−1]P_i = [x_0, \dots, x_{i-1}] (x0x_0 is the anchor) via addressable prefix memory and the last 64 strictly verified target features. Once slot ii repairs C to D (Fig. 3), all subsequent decisions condition on the actual prefix containing D.
  2. Conservative KEEP/REPAIR decomposition. Corrupting a correct early native token truncates the entire subsequent prefix. DRelay separates ranking alternatives (Rank, over k>0k > 0 only) from deciding whether to override the native choice (Gate): Gate starts from the native pool-mass log-odds prior (Appendix Eq. 6) with zero-initialized residuals, triggering a repair only when gi+log⁡rik∗>0g_i + \log r_{ik^*} > 0.
  3. Prefix-value-weighted joint training. Section 5 jointly trains the backbone and selector on mixed reachable boundaries (scout) and uniform anchors, weighting the repair loss by reach ×\times continuation value (Eq. 5) to prioritize early repairs that unlock long continuations.

Overall, DRelay's modest extra overhead buys more accurate cross-position repairs, longer accepted prefixes, and higher average end-to-end speedup.

W2 / Q2: Why Global Read Requires Both Top-16 Slot Mixing and Candidate Reads

Let vikv_{ik} be the top-KK candidates at slot ii, eike_{ik} their embeddings, and πik\pi_{ik} the pool-renormalized draft probabilities. Setting K=16K=16 follows Fig. 2 (Left): at the first error in base predictions, the original top-16 pool covers the target token in 94.0% (DFlash) and 95.1% (Domino) of cases.

The two stages work at complementary granularities while keeping compute small:

  • Slot mixing (H=15H=15 slot nodes, 2 bidirectional attention–FFN blocks): While the noncausal backbone performs initial fusion across masked slots, it has not yet directly read or interacted over output candidate distributions (Sec. 4.2). This stage mixes slot features (draft hidden states and confidence statistics) with soft summaries ∑kπikeik\sum_k \pi_{ik} e_{ik} into block context uiu_i. Because uiu_i is shared across all KK candidates at slot ii, weighted averaging blends competing future continuations and cannot tell which candidate matches which continuation.
  • Candidate reads (H×K=240H \times K = 240 candidate nodes, 1 attention layer): Each candidate forms node nik=LN(ui+Wneik)n_{ik} = \mathrm{LN}(u_i + W_n e_{ik}) and queries all 240 nodes with draft prior log⁡qj(vjl)\log q_j(v_{jl}) and relative bias bj−ib_{j-i} (Eq. 1). If slot ii has "a/an" and slot i+1i+1 has "apple/banana/orange", "an" and "a" attend selectively to "apple/orange" vs. "banana", obtaining distinct evidence GikG_{ik}.
  • Why the 15→24015 \to 240 two-stage design: Slot mixing alone lacks candidate-level discrimination; candidate reads alone lack cross-position context uiu_i, and one FFN-free attention layer cannot model multi-step block structure. Conversely, multi-layer attention–FFN over 240 nodes would enlarge the attention matrix by (240/15)2=256×(240/15)^2 = 256\times. Mixing on 15 slots then reading once over 240 candidates keeps Global Read at 2.25M parameters, run once per block; outputs are scaled by λi\lambda_i with a zero-valued null key/value (Sec. 4.2) to suppress uncertain context.

Isolating ablation. Under Table 3's setup (Qwen3-4B, T=0T=0, H=15H=15, K=16K=16, 8-benchmark mean; Sec. 6.3), we trained two variants from scratch, each removing one stage:

Variantτ\tau (8-benchmark mean)
Full7.58
Slot mixing only (w/o candidate reads)7.32
Candidate reads only (w/o slot mixing)7.21
No Global Read (Table 3)7.16

Over No Global Read (7.16), slot mixing alone adds +0.16 and candidate reads alone adds +0.05, while combining both adds +0.42—twice their sum (0.21)—confirming complementarity: without block context from slot mixing, single-layer candidate reads barely help; with slot summaries alone, over half the gain is lost.

W3 / Q3: Parameter Count, Inference Latency, Training Cost, and Convergence

Parameters. The 35.06M selector on Qwen3-4B (Sec. 4.3) breaks down as follows (rows rounded independently):

ModuleParams (M)Share
Global Read2.256.4%
Slot mixing (input proj. + 2 bidirectional attention–FFN)1.785.1%
Candidate reads (candidate-specific attention, Eq. 1)0.260.7%
Output projections and fusion scalar λi\lambda_i0.200.6%
Causal Selector32.8193.6%
Slot and candidate input projections3.189.1%
Shared hidden residual readout (2-layer wide MLP)11.5432.9%
Rank branch4.0711.6%
Gate branch1.855.3%
Verified-history read (last 64 tokens)1.464.2%
Prefix memory and latest chosen token conditioning5.4415.5%
Anchor-feature projections5.2815.1%
Total35.06100%

Global Read totals 2.25M; most parameters sit in width-2560 projections. Parallel drafting invokes the frozen target LM Head once (0.29 ms, Fig. 4) to get top-16 pools; inside the selector, the shared residual MLP (11.54M) predicts a 2560-d vector and scores only the 16 frozen candidate embeddings via inner products, without repeating a full-vocabulary projection across the 15 causal steps (Sec. 4.3). On Qwen3-4B, head sizes are 1.97M (DFlash2), 32.04M (LiLiCorr; the 36.76M noted by the reviewer is its 8B size), and 35.06M (DRelay, same range as LiLiCorr).

Inference overhead. Static projections, Global Read, and history reads run once before the 15-step loop, and the selector executes as one CUDA Graph (FP32 weights ≈0.13\approx 0.13 GiB, reusing frozen target embeddings). On Qwen3-4B + SGLang + H800 (W1 table), round latency is 6.38 ms (DFlash), 6.62 ms (+3.8%, DFlash2), 6.65 ms (+4.2%, LiLiCorr), and 6.93 ms (+8.6%, DRelay); the ≈0.6%\approx 0.6\% and ≈2.8%\approx 2.8\% in their papers reflect different setups. DRelay adds ≈0.3\approx 0.3 ms over both, but its ≈11.4%\approx 11.4\% higher τ\tau yields faster decoding (3.57× vs. 3.39×/3.38×) and exceeds its entire 8.6% latency overhead over DFlash.

W3 / Q3 (cont.): Training Cost and Convergence

Training cost. Like the baselines, DRelay trains on 1.42M PerfectBlend across eight H800 GPUs via SpecForge (Sec. 6.1), with incremental cost from three sources: (1) single-stage joint training adds no extra training stage and no extra target forward pass (reusing target hidden features already captured for drafter training); (2) atop the ≈0.5\approx 0.5B 5-layer parallel draft backbone, the 35.06M selector adds ≈7%\approx 7\% trainable parameters (≈0.6%\approx 0.6\% above LiLiCorr), with FP32 selector weights and optimizer states under 0.6 GiB; and (3) each step adds only the selector's forward/backward pass and one no-gradient scout forward pass to locate reachable block boundaries (Sec. 5).

Convergence. Zero-initialized terminal residuals combined with the Appendix Eq. 6 prior recover the pool-renormalized native distribution at initialization (Sec. 4.3), simplifying the repair rule to gi+log⁡rik∗=log⁡qi(vik∗)−log⁡qi(vi0)≤0g_i + \log r_{ik^*} = \log q_i(v_{ik^*}) - \log q_i(v_{i0}) \le 0 so step 0 is strictly KEEP at every slot. During training, the selector loss coefficient αt\alpha_t ramps from ≈0.05\approx 0.05 to 1, while prefix-value weights activate progressively with stopgrad and an upper bound (Sec. 5). Qwen3-4B, 8B, 3.5-4B, and all ablation runs converged smoothly in a single stage under identical hyperparameters.

W4 / Q5: Ablation Setup and Cross-Scenario Performance

Experimental setup. Every entry in Tables 3 and 4 is already the arithmetic mean across all eight benchmarks in Table 1, rather than a single dataset (Sec. 6.3). The full protocol is: Qwen3-4B, T=0T=0, H=15H=15, K=16K=16, single H800, SGLang; Full (7.58) is the corresponding row in Table 1; and every row in Tables 3–4 as well as the new W2 variants was trained independently from scratch on 1.42M PerfectBlend with identical hyperparameters. We will add these details to the captions of Tables 3–4 and report per-benchmark numbers in the appendix.

Cross-scenario performance. Combining Table 1 with the per-benchmark τ\tau in W1 (Qwen3-4B, T=0T=0) shows clear complementarity across math, code, and chat:

  • Causal prefix conditioning only (Domino, DSpark): large gains on GSM8K (≈+2.8\approx +2.8) and chat (+0.8–0.9), but flat or degraded on AIME25 (−0.07/+0.15-0.07/+0.15) and LCB (−0.51/−0.33-0.51/-0.33).
  • Block-wide reselection only (DFlash2, LiLiCorr): more uniform gains (+0.29–1.21), outperforming Domino/DSpark on AIME25 and LCB, but lagging behind on step-by-step prefix-sensitive GSM8K and chat.
  • DRelay (combining both): Global Read resolves early ambiguities using later block context while Causal Selector propagates repairs along the actual prefix, outperforming all five baselines on every benchmark (+0.83–3.57 over DFlash; +0.19–0.73 over the strongest baseline per benchmark).

Q4: Linear/Hybrid-Attention and MoE Target Models

We evaluated DRelay on the hybrid-attention target model Qwen3.5-4B, whose backbone combines linear-attention and standard attention layers, with no architectural or hyperparameter changes (H=15H=15, K=16K=16, identical selector design and training recipe as Qwen3-4B). Per-benchmark τ\tau at T=0T=0:

MethodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAlpacaAvg.
DFlash6.567.787.436.536.117.324.203.356.16
DRelay9.859.698.237.827.068.315.254.367.57

DRelay outperforms DFlash on all eight benchmarks (+0.80 to +3.29), raising mean τ\tau from 6.16 to 7.57 (+1.41, +22.9%), matching the gain on dense Qwen3-4B (6.15 →\to 7.58, +1.43). By interface, the selector consumes only the parallel drafter's hidden states and top-KK distributions, frozen token embeddings, and target hidden features at verified positions; internal attention operators or MoE routing in the target do not alter any computation in Global Read or Causal Selector.

We did not evaluate MoE target models during the rebuttal period. Adapting parallel drafting to a new large MoE model requires training a matched parallel drafter from scratch and completing serving integration, which exceeded our multi-GPU budget within the rebuttal window (neither DFlash2 nor LiLiCorr reports MoE results in their papers).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.