- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
DRelay addresses a limitation of speculative decoding: an early draft error can prevent later correct tokens from being accepted. It uses a Global Read module to gather candidate-specific information across the draft block, then a Causal Selector to keep or replace each token based on that information and the already selected prefix. The draft model and selector are trained jointly, with more weight on decisions that can extend the accepted prefix.
Critical strengths
S1. Their motivation is interesting and compelling: in SD's drafting, even if suffix draft tokens are correct, early token's rejection forces them all to be rejected. By correcting one or a few early rejection may increase the number of accepted tokens by much, which is interesting (Section 3.3).
S2. In the evaluation they included accuracy ablation across the loss proposals, which is interesting.
Critical weaknesses
W1. There is insufficient evidence that early errors can be corrected with the suffix's information. In other words, the meaning of Section 3.2 and Figure 2 is unclear. The reported log probability is calculated using the target model's information, making the experiment meaningless: it does not rule out at all that the accuracy improvement comes from the target model rather than from information added through the suffix. It should be changed to use the draft, and the experimental method should also be changed. For example, the authors should also test whether the probability decreases when the same method is applied to samples where the suffix is incorrect.
W2. The paper does not explain theoretically or experimentally why the proposed Global Read/Causal Selector approach can reduce early-token errors compared with existing methods that predict each token. Sections 4.2, 4.3, and 5 mainly describe how the proposed method works, but do not explain theoretically why it helps reduce early-token errors. Even the method itself is difficult to understand from the paper's descriptions.
Regarding this, I suggest experiments that seem necessary but are missing. The paper should separately report the frequency of REPAIR and an extended confusion matrix of selector decisions, with both probabilities and raw counts (e.g., 10/15), for the following cases: (1) correct native token → KEEP, (2) correct native token → REPAIR, (3) wrong native token → KEEP, (4) wrong native token → REPAIR with the correct token, and (5) wrong native token → REPAIR with another wrong token. Correctness should be defined against the target greedy reference. The paper should also report the average number of correct suffix tokens accepted when a correction is successful.
W3. The presentation needs significant improvements. The logical progression is unclear (the paper uses many unnecessary qualitative and figurative sentences and terms, and does not sufficiently connect its claims—for example, line 269: "A single summary can obscure competing continuations."). In addition, terms newly used in the paper (e.g., "addressable prefix memory," "candidate pool") are used without definition, and the variables in the equations (e.g., i, j, k, l, and g in Sections 4.2 and 4.3) are not sufficiently explained.
W4. The paper applies several heuristics (e.g., setting k to 16 when selecting top-k candidates in Global Read, and considering the past 64 tokens in the Causal Selector). It needs at least a brief justification for how these values were chosen.
Questions for authors
W1--W4 and following.
Q1. How much of the performance improvement comes from retraining the backbone rather than from repair?
Q2. If the correct answer is in the top-16 candidate pool in about 94% of cases, a simpler alternative would be to have the target directly verify the pool candidates using tree verification (e.g., EAGLE-style methods or TAPS), rather than having the selector choose a single path. What advantages does DRelay offer over this approach?
7 comments
Response to Reviewer s1Mz
W2: Why Global Read and the Causal Selector Can Reduce Early-Token Errors
The paper already provides three levels of evidence.
First, the correct token remains in the top-16 candidate pool at the first error position in 94.0% of DFlash cases and 95.1% of Domino cases, and in 88.8% of Domino's actual served traces. This indicates that many failures arise from candidate selection rather than candidate generation.
Second, when the correct candidate is available and the subsequent draft continuation is correct, the continuation provides discriminative information for identifying the earlier correct candidate (46/86 for DFlash and 31/69 for Domino).
Third, the one-edit oracle increases accepted length by 1.65 and 1.22 tokens for DFlash and Domino, respectively, demonstrating that correcting a single early position can recover a subsequent run of correctly predicted tokens.
The two components of Global Read also serve distinct purposes.
Slot mixing first compresses the candidate distribution at each position into a summary and then mixes these summaries across positions to construct shared context for the entire block. However, candidates C and D at the same position share this summary, which alone cannot determine whether C or D should be selected.
Candidate reads then construct separate representations for individual candidates. Each candidate uses its own query to attend to candidates across the entire block, allowing C and D to retrieve different continuation evidence.
The Causal Selector subsequently makes causal decisions using a strictly maintained selected-prefix memory. If C is repaired to D at the current position, the next position conditions on D rather than the original C.
In other words, Global Read addresses how earlier candidates can exploit later information, while the Causal Selector addresses how a repair is propagated to subsequent positions.
The Full versus No Global Read ablation shows that accepted length decreases from 7.58 to 7.16 when Global Read is removed, directly validating the contribution of this information pathway.
Reducing DRelay to "independently predicting one token at each position" is inconsistent with both the equations and the implementation presented in the paper.
Regarding the reviewer's requested five-category KEEP/REPAIR statistics, we disagree that they are necessary evidence for establishing the validity of the method.
These statistics are post-hoc classifications of actions already taken by the selector. The overall REPAIR frequency can be obtained by summing the three corresponding categories, and raw-count examples such as "10/15" do not themselves provide additional mechanistic evidence. The correctness reference is already clearly defined as the target-greedy token under the same target prefix.
Likewise, the average number of accepted suffix tokens following a successful repair is a conditional decomposition of accepted length and cannot replace full-system metrics.
If included, these statistics should serve as supplementary diagnostics rather than as prerequisites for validating Global Read or grounds for dismissing the existing evidence.
W3: Terminology, Variables, and Readability
Both "candidate pool" and "addressable prefix memory" are standard concepts in speculative decoding, and the paper provides concrete definitions specific to our method.
Section 4.1 defines the candidate pool as the top-16 candidates retained at each position.
Section 4.3 defines the prefix memory as an addressable storage structure containing the anchor and previously selected entries, with a strict (j<i) attention mask.
The variables (i), (j), (k), (l), and (g) denote the current position, a previous position, the candidate index, the attended candidate node, and the gate logit, respectively.
Therefore, the reviewer's claim that these concepts are undefined is factually inaccurate.
At most, the definitions are distributed across two subsections, which constitutes a presentation issue rather than an undefined algorithm.
We will consolidate the notation and provide a clearer four-step description of the execution process. These improvements do not require any modification to the proposed method.
W4: The Choice of K=16 and the 64-Token History Window
K=16 represents an operating point balancing candidate coverage against selector overhead. The reported coverage results already demonstrate that this candidate pool retains the vast majority of potentially repairable tokens.
Increasing K incurs additional candidate-reading computation and memory overhead, whereas decreasing K risks excluding correct candidates.
The 64-token history window provides a bounded read over strictly verified history. Its size affects computational cost and performance, but it is not a prerequisite for the correctness of the proposed method.
Neither parameter is claimed to be a theoretically optimal constant across all models or workloads.
Response to Reviewer s1Mz
Q1: Does the Performance Improvement Come from Backbone Retraining or Repair?
The "No selector loss" configuration retains the draft objective while disabling repair, achieving an accepted length of 7.22, compared with 7.58 for the full model.
This already demonstrates an additional improvement of 0.36 tokens associated with the complete repair-enabled design. Therefore, the entire performance improvement cannot simply be attributed to backbone retraining.
A more detailed decomposition of frozen-backbone versus jointly trained-backbone effects would require an additional 2×2 factorial experiment.
However, this is a different question from whether repair provides additional benefits. The existing ablation already addresses the latter.
Q2: Why Not Directly Verify the Top-16 Candidate Pool Using Tree Verification?
The reviewer conflates candidate coverage at a single position with coverage of a complete target-conditioned verification tree.
The reported 94.0% only indicates that the correct token appears in the top-16 marginal candidate pool at a particular error position. It does not imply that subsequent positions already have target-model states conditioned on that token, nor does it guarantee that a complete correct path exists in the verification tree.
Verifying multiple branches requires the target model to compute nodes under different prefixes and maintain the corresponding KV-cache states. This changes both target-side computation and memory requirements.
In contrast, DRelay first uses draft-side candidate information to select a single path and then performs conventional single-path target verification.
Therefore, tree verification is not an equivalent or inherently simpler replacement for DRelay.
A fair comparison must match the target checkpoint, hardware, inference backend, number of target-scored nodes or FLOPs, and end-to-end latency.
Our claims are explicitly limited to single-path parallel speculative decoding. We do not claim that DRelay outperforms tree-based verification under every possible verification budget.
Response to Reviewer s1Mz
We thank the reviewer for their comments. We are willing to consolidate the notation and clarify the experimental descriptions. However, we disagree with treating several misinterpretations of our offline diagnostics, methodological mechanisms, and tree verification as fundamental soundness concerns.
W1: Whether Figure 2 Demonstrates That Suffix Information Can Repair Early Errors
The purpose of Figure 2 is not to simulate the selector at deployment, but to investigate a more specific question: Does later draft content contain discriminative signals that can distinguish between earlier candidates?
The real suffix and control suffix use the same frozen target model, identical prefixes, current-position candidates, suffix lengths, and correctness conditions. The only difference is whether the suffix is aligned with the original continuation. Moreover, the target probability of the current-position candidate is explicitly excluded from the score. Therefore, the target model serves only as a common offline assessor in both groups and cannot, by itself, explain the observed difference.
Replacing the score with draft-model probabilities is not a valid correction to this experiment. Draft-model probabilities measure the draft model's internal consistency, whereas Figure 2 measures the compatibility between candidate tokens and the target continuation. These are fundamentally different questions.
With real suffixes, the correct candidate is selected in 46 out of 86 DFlash cases and 31 out of 69 Domino cases, representing improvements of 53.5 and 44.4 percentage points over the matched controls, respectively.
At inference time, DRelay does not access these target-model scores, future target tokens, first-error locations, or oracle filtering conditions. Figure 2 only establishes that later draft candidates contain exploitable cross-position signals. The practical benefit at deployment is independently validated by the Full versus No Global Read ablation.
Therefore, the reviewer's conclusion that this experiment is "meaningless" is unjustifiably strong. Deliberately constructing incorrect suffixes could serve as an additional negative control, but it is not a prerequisite for the validity of the existing diagnostic.