acceptodds
Under review as a conference paper at ICLR 2027

ReDraft: Your Verifier Is Secretly a Depth Supervisor for Recurrent Speculative Decoding

Abstract

Speculative decoding accelerates LLM inference by having a cheap drafter propose token blocks that the target model verifies in parallel. Yet drafter depth is typically fixed by ablation and applied uniformly across blocks, despite substantial variation in draft difficulty: a per-block depth oracle gains accepted tokens over the best fixed depth, while the cheapest depth suffices for 40% of blocks. Standard halting heuristics fail to exploit this variation: convergence- and confidence-based exits collapse to maximum depth, reducing throughput by 20-27%, and their tuned thresholds transfer poorly across tasks, model scales, and training data. We instead exploit a supervision signal unique to speculative decoding: the verifier reveals each draft's accepted length without additional model computation. We introduce ReDraft, a learned policy that selects drafter depth for each block. ReDraft uses a -layer weight-shared core iterated to variable depth and a lightweight head that predicts each block's acceptance curve from verifier-derived labels. A renewal-reward stopping rule converts these predictions into depth choices using measured hardware costs, while leaving verification—and thus losslessness—unchanged. ReDraft plans once per block with only 1.8% overhead, and a regret bound shows that throughput loss depends on errors in the relative values of candidate depths. Across five benchmarks and three targets spanning two model families, a single recipe yields tuning-free policies within 4-6% of the per-task tuned optimum, with no statistically significant difference on GSM8K, while allocating computation similarly to the oracle. The recurrent drafter achieves a lossless speedup on GSM8K, matching the released -layer DFlash drafter with 40% of its parameters and half its training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, recurrent drafting, adaptive computation, test-time compute, efficient LLM inference, verifier supervision, learned inference policies, lossless inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R6BzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies adaptive drafting depth in speculative decoding. It proposes ReDraft, which replaces the fixed five-layer drafter in DFlash with a two-layer weight-shared core trained with varying iteration counts and supervision of intermediate outputs. A lightweight predictor is trained on verifier-derived accepted lengths collected offline. After the first drafting iteration, it predicts the expected accepted length at each candidate iteration count. A throughput-based rule combines these predictions with measured hardware costs to select the total iteration count once per block. The drafter then runs to that count without further decisions, while target verification remains unchanged. The paper also bounds throughput loss in terms of errors in predicted acceptance differences between depths. Experiments cover five benchmarks with Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct targets.

Critical strengths

S1. The motivation is convincing. The analysis shows why internal convergence alone can be misleading, providing a clear rationale for using verifier feedback to allocate drafting computation.

S2. The evaluation is systematic. Controlled ablations and policy comparisons separate the benefits of recurrent training from those of adaptive depth selection, supporting the main design choices rather than only reporting overall speedups.

S3. The approach offers a practical alternative to task-specific depth tuning. It approaches tuned fixed-depth performance on several benchmarks while avoiding the overhead of repeated stopping decisions.

Critical weaknesses

W1. The evaluation is difficult to follow because the main comparisons are not consolidated. For example, the DFlash comparison, cross-model results, and predictor diagnostics are spread across paragraphs and several appendices. This makes it difficult to separate the benefits of recurrent drafter training from those of depth selection, and to track which results use comparable settings. Compact main-text summaries and clearer labels for each experiment would make the evidence easier to assess.

W2. The novelty and practical benefits relative to DSpark (arXiv, 2026) are not clearly established. DSpark learns acceptance estimates from draft–target distributions and combines calibrated confidence predictions with hardware-aware throughput optimization. I know that ReDraft controls recurrent depth rather than verification length, but I believe the paper should discuss or evaluate this closely related approach.

W3. The scaling behavior of the predictor needs further analysis. Appendix H reports that agreement with the best depth under the index rule falls from 69% on Qwen3-4B to 53% on Qwen3-8B, while Appendix I finds little reduction in the throughput gap from doubling the label budget. Since the evaluated targets stop at 8B, the cause of this trend and its implications for larger models remain unclear.

W4. The basis for first-iteration acceptance prediction and the choice of predictor inputs need further evaluation. ReDraft predicts target acceptance at later recurrent depths before those drafts are generated or verified. It is unclear what properties of the mean-pooled drafter state and sorted per-position top-token probabilities make this prediction possible. The rationale for choosing these inputs over alternatives is also unclear.

W5. Figure 2 makes the execution flow difficult to follow. The relationship between token positions, recurrent iterations, and successive draft–verify rounds is unclear. For example, why are x2x_2 and x3x_3 omitted, and why do the labels h5h_5 and h6h_6 appear in consecutive rounds while h1h_1, h2h_2, h3h_3 doesn’t? The meaning of these indices and their relationship to the selected depth should be clarified.

Questions for authors

Q1. Please consolidate the core comparisons in the main text, particularly the DFlash results and the 4B/8B comparison. Compact tables or figures should distinguish drafter-training gains from depth-policy gains and clearly identify differences in training and evaluation settings. Short descriptive paragraph headings would also help readers follow the separate experiments.

Q2. Why was DFlash chosen as the foundation without a comparison to DSpark? Please clarify what the recurrent drafter and first-iteration depth policy add beyond DSpark. A direct comparison on shared target models is needed to establish the practical benefit. If the methods are complementary, could you discuss whether ReDraft would provide additional gains when applied to a DSpark-style drafter?

Q3. What explains the weaker depth selection at 8B, and why does doubling the verifier-labeled data used to train the predictor not substantially reduce the throughput gap relative to the best fixed-depth policy? How do you expect this behavior to change for larger targets? Larger-model experiments may be difficult to complete during rebuttal. Nevertheless, evaluation on at least one target above 8B is necessary to substantiate scalability beyond the evaluated range and should be included in a revised version.

Q4. Why can the first iteration’s output be used to predict token acceptance at later recurrent depths? Why were the mean-pooled drafter state and sorted per-position top-token probabilities chosen as predictor inputs? Could you evaluate removing each input, adding already available target hidden states, or using those states in place of the drafter state?

Q5. Please revise Figure 2 or expand its caption. Clarify the hidden-state indices and distinguish recurrent iterations within a block from transitions between blocks. Also show what information the predictor takes as input and how predicted accepted lengths and measured execution costs are combined to select the depth.

8 comments
AuthorsOpenReview ↗

Response to Reviewer R6Bz (1/3)

We thank the reviewer for the detailed review and agree that the drafter, policy, and training-data effects should be separated, and that the relation to DSpark should be explicit. We make both distinctions in the revised Section 5.

W1 / Q1: consolidating the main comparisons. We now report the main Qwen3-4B comparisons in one table (Table 3), evaluated at temperature 0 on the same hardware and harness with 200 prompts per benchmark. Speedup is relative to autoregressive decoding, and acceptance length appears in parentheses:

comparisondrafter (training data)depthGSM8KMATH-500HumanEvalMBPPMT-Bench
released systemDFlash-5, 5 distinct layers (own data)fixed4.39× (6.08)5.36× (7.38)4.69× (6.43)4.19× (5.64)2.43× (3.20)
training datarecurrent 2-layer core (main corpus + 120K code prompts)best fixed per task3.75× (4.95)3.98× (5.40)3.18× (4.49)2.98× (3.95)1.98× (2.55)
policyrecurrent 2-layer core (main corpus)best fixed per task3.87× (5.34)4.01× (5.50)2.61× (3.74)2.65× (3.55)1.96× (2.52)
policyrecurrent 2-layer core (main corpus)plan-once3.81× (5.21)3.96× (5.37)2.62× (3.70)2.61× (3.58)1.97× (2.51)

These rows separate three effects:

  1. Drafter architecture: the recurrent two-layer core compared with five distinct layers under matched data and training recipe. The five-layer drafter has 1.5–6.7% lower mean throughput.
  2. Depth policy: the best fixed depth per task compared with one plan-once policy on the same recurrent drafter. Plan-once comes within 1.7% without per-task tuning.
  3. Training data: the released DFlash-5 compared with our recurrent drafter under their respective corpora. Additional code data substantially narrows the gap on the code benchmarks.

The revision moves a compact version of this comparison into Section 5 under descriptive headings, so these effects are no longer distributed across separate sections. Section 5.4 also places the 4B, 8B, Llama, and 14B results together. The consolidated table's caption states that its policy rows use the retrained drafter and predictor, which differ from Table 1.

AuthorsOpenReview ↗

Response to Reviewer R6Bz (2/3)

W2 / Q2: relation to DSpark, and whether ReDraft adds value on a DSpark-style drafter. DSpark and ReDraft adapt different quantities. DSpark adapts how many drafted tokens to verify, whereas ReDraft adapts how much computation the drafter spends on each block. DSpark uses a parallel block drafter together with a lightweight sequential module and schedules verification length using estimated prefix-survival probabilities and engine throughput. Its drafter runs at a fixed depth. Both methods learn acceptance estimates and optimize throughput with measured costs, but they make different decisions: DSpark estimates how far each drafted prefix will survive to choose how many tokens to verify, whereas ReDraft predicts how acceptance changes with additional drafting computation. ReDraft therefore requires a drafter whose computation can vary at inference. Our recurrent two-layer core provides this depth axis, and the acceptance-curve predictor selects the depth using measured drafting and verification costs. We build ReDraft on DFlash because its released drafters and training recipe let us compare recurrent and distinct-layer drafters under matched conditions, isolating recurrent depth from unrelated training differences. We also test depth allocation on a DSpark-initialized core. For Qwen3-14B, we initialize our recurrent core from two layers of DSpark's released 14B backbone and train it with our recipe. On this core, plan-once remains close to the fixed-depth frontier: it comes within 0.9–1.5% of the best fixed depth per benchmark in wall-clock decoding and is 1.7% faster than the best global depth in exact replay. Under the 0.25–4× drafting-cost sweep in replay, plan-once matches or exceeds the best global depth at each cost, by up to 6.3%, while the global depth that is best at the measured cost (r=2r=2) loses up to 5.9% and r=3r=3 loses up to 13.9%.

For an end-to-end reference point, we also run the released DSpark-14B checkpoint on the same prompts. In vLLM, DSpark-14B reaches 5.05×, 5.11×, 4.60×, 4.33×, and 3.09× over vLLM's autoregressive decoding on GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench, respectively. Plan-once in our research harness reaches 3.06×, 3.04×, 2.01×, 2.06×, and 1.64×. Because these results use different engines, they do not provide a controlled end-to-end comparison. We therefore also compare accepted tokens per verification step, which more directly isolates draft quality from serving-engine throughput: 6.50, 6.46, 5.78, 5.52, and 3.92 tokens for DSpark, compared with 4.15, 4.07, 2.74, 2.72, and 2.15 for our recurrent core. DSpark is faster end to end, and we do not claim that ReDraft subsumes DSpark. DSpark uses a separately trained drafter, a sequential head, its own training data, and a verification scheduler, whereas our recurrent core has two shared layers and uses our training corpus of about 240K regenerated Qwen3-14B sequences. Our controlled claim is narrower: depth allocation remains useful on a recurrent core initialized from the DSpark backbone. The two methods operate on different axes and could potentially be combined, with DSpark's verification-length scheduling applied together with ReDraft's drafting-depth control. We leave this combination to future work.

AuthorsOpenReview ↗

Response to Reviewer R6Bz (3/3)

W3 / Q3: weaker depth ordering at 8B, and a target above 8B. We retrain the Qwen3-8B drafter and predictor with the same protocol we use at 4B and 14B. In wall-clock decoding, plan-once now comes within 0–1.9% of the best fixed depth on each benchmark, compared with 4–5% in the original submission, and it is 2.2% faster than the best global depth in exact replay. Its depth ordering also matches the other targets: Kendall τ\tau is 0.65 at 8B, compared with 0.59 at 4B and 0.63 at 14B under the same protocol. These values are not directly comparable to the paper's held-out 0.74 and 0.60 because those results use a different evaluation set. We also train a new drafter and predictor for Qwen3-14B. Plan-once comes within 0.9–1.5% of the best fixed depth per benchmark in wall-clock decoding and is 1.7% faster than the best global depth in exact replay, while the per-block oracle remains 12–16% faster than the best fixed depth.

The weaker ordering in the original 8B experiment therefore does not appear to result from target size alone. In that experiment, the pairwise-difference error, which enters the regret bound directly, increased from 0.52 to 0.87 tokens, and doubling the verifier-labeled training set did not materially reduce the gap. The retrained pipeline changes both the regenerated training data and the depth-selection rule, so we do not isolate which change closes the gap. The feature ablation shows the same pattern at 8B as at 4B and 14B: removing the pooled drafter state reduces the replay gain from 2.2% to 0.6%. The revision reports the retrained 8B results in Section 5.4 and Appendix J.

W4 / Q4: why iteration-1 features predict later acceptance, and which inputs matter. After the first recurrent iteration, the drafter has produced a complete draft for all 16 positions. The pooled hidden state summarizes the block context and current draft, while the per-position maximum probabilities indicate where the draft is likely to break. Later recurrent iterations refine this draft, so these first-iteration signals provide information about where additional computation is likely to help. The predictor ablation supports this interpretation. Removing the pooled drafter state removes most of the mixed-workload gain at both 4B and 14B, whereas removing the sorted per-position probabilities has a smaller effect. At 4B, adding the target hidden state, or replacing the drafter state with it, does not improve the predictor. The deployed policy can therefore make its depth decision from information the drafter already produces, without additional target-model internals.

W5 / Q5: Figure 2. We have redrawn Figure 2 to separate block-position indices from sequence-position indices across speculative rounds and to make the execution flow explicit. Each circle denotes the drafter state of one position in the current block, and rr denotes the number of applications of the shared two-layer core. The predictor reads the iteration-1 states and per-position top-token probabilities, predicts the acceptance curve, and selects i∗=arg⁡max⁡i[a^(i)+1−λ∗c(i)]i^\ast=\arg\max_i[\hat a(i)+1-\lambda^\ast c(i)], where c(i)=i td+tvc(i)=i\,t_d+t_v and tdt_d and tvt_v are measured at startup. The drafter then runs open loop to i∗i^\ast, followed by target verification. We index positions contiguously within each draft–verify round and distinguish recurrent iterations within a block from successive speculative rounds. Each round ends with the target token, and a rejected position is drafted again in the next round. In the original figure, the circles carried token indices from different rounds, so x2x_2 and x3x_3 appeared to be missing and the indices did not line up across rounds. In the redrawn figure, hjh_j, xjx_j, and yjy_j denote the drafter state, drafted token, and target token at sequence position jj. Each round starts at the first uncommitted position, so positions after a rejection are drafted again, as h2h_2 and h3h_3 are in the second round. The revised caption states these distinctions explicitly.

Reviewer_aATDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

ReDraft addresses the fixed drafting depth problem in speculative decoding. It replaces the fixed-depth parallel drafter with a 2-layer weight-shared recurrent drafter and uses verifier-derived acceptance curves to predict the value of additional recurrent computation for each block, enabling adaptive depth selection. The key insight is that the verifier can serve as a depth supervisor. Overall, the problem is important, the idea is clear and memorable, and the experiments largely support the main claims.

Critical strengths
  1. Verifier feedback is naturally connected to the value of additional drafting computation, leading to a simple and well-motivated adaptive-depth framework. This provides a principled basis for making block-wise depth decisions based on predicted acceptance.
  2. The plan-once policy avoids the substantial overhead of closed-loop decisions while retaining most of the adaptive benefit. This makes the proposed approach more suitable for practical deployment with limited decision-making overhead.
  3. The paper includes recurrent-depth ablations, predictor evaluation, multiple models/tasks, temperature settings, matched-data comparisons, and multi-seed results. These experiments provide broad evidence for the effectiveness and robustness of the proposed approach.
Critical weaknesses
  1. Multiple experimental conclusions are presented together in the same paragraph, while broad references to appendix results make it difficult to clearly distinguish the findings of different experiments. The presentation could be improved by better organizing and separating the experimental discussions.
  2. ReDraft has an average depth of 2.7 with a 2-layer recurrent core, corresponding to roughly 5.4 layers of computation, which is comparable to DFlash’s 5-layer drafter. A direct comparison of drafting latency, acceptance quality, and end-to-end throughput would clarify whether ReDraft’s advantage comes from better draft efficiency, better draft quality, or both.
  3. The predictor uses hidden states, per-position maximum probabilities, and iteration embeddings, but no feature-level ablation is provided. Ablating these features individually would clarify which signals are essential to effective acceptance-curve prediction.
Questions for authors
  1. Could the authors provide a direct comparison of drafting latency, acceptance length, and end-to-end throughput between ReDraft and the 5-layer DFlash drafter under matched settings? This would help clarify whether ReDraft’s speedup mainly comes from improved draft efficiency, better draft quality, or both.
  2. Could the authors provide feature-level ablations by removing the hidden-state features, per-position maximum probabilities, and iteration embeddings individually? This would clarify the relative contribution of each signal and identify which features are essential for effective acceptance-curve prediction.
7 comments
AuthorsOpenReview ↗

Response to Reviewer aATD (1/2)

We thank the reviewer for the constructive review and for raising two questions that call for direct comparisons: how ReDraft's recurrent drafter compares with the five-layer DFlash drafter, and which predictor inputs drive the gain. We add both analyses and clearly separate the controlled architectural comparison from the comparison with the released checkpoint.

W1: organization of the experimental discussion. We reorganized Section 5 so that each experiment now appears under a descriptive heading, including "Plan-once against one global depth on a mixed workload" and "Separating the drafter, the policy, and the training data." We also consolidate the main Qwen3-4B comparisons in Table 3. Each row identifies the drafter and its training data, and the caption specifies the evaluation protocol. Throughout the section, we now cite the specific table supporting each claim instead of referring broadly to the appendix.

W2 / Q1: direct comparison with the five-layer DFlash drafter. We run the released DFlash-5 checkpoint in our harness on the same hardware. Table 3 reports speedup, acceptance, and training data for each configuration. On GSM8K, DFlash-5 reaches 4.39×, compared with 3.87× for our recurrent drafter at its best fixed depth and 3.81× for plan-once. DFlash-5, however, was trained on a different and larger corpus, so this comparison does not isolate the architectural effect. We therefore make the controlled comparison separately. With the same training data and recipe, the matched five-layer drafter has 1.5–6.7% lower mean throughput than the recurrent core across the five benchmarks (7–10 seeds). Drafting latency is also similar: one DFlash-5 pass takes about 5.2 ms, compared with 4.7 ms for two recurrent iterations and 6.5–6.7 ms for three, depending on context length. Thus, the matched five-layer drafter does not provide a substantial latency advantage, and its lower throughput coincides with lower acceptance on four of the five benchmarks (Appendix I).

The recurrent drafter uses 46% as many total parameters as the matched five-layer drafter and, unlike a fixed-depth drafter, exposes depth as a variable at inference. These results also distinguish draft efficiency from draft quality. At the two to three iterations that plan-once typically selects, the recurrent core performs roughly as much computation as one DFlash-5 pass and takes 4.7–6.7 ms, compared with 5.2 ms for DFlash-5. The gain therefore does not come from simply making drafting cheaper. Under matched training, the recurrent core achieves higher acceptance at comparable drafting cost and can spend less computation on easy blocks and more on hard ones. The gap to the released DFlash-5 checkpoint also narrows with additional code data: HumanEval improves from 2.61× to 3.18× and MBPP from 2.65× to 2.98×. Section 5 now distinguishes the controlled architectural comparison from the released-checkpoint comparison so that these two effects are not conflated.

AuthorsOpenReview ↗

Response to Reviewer aATD (2/2)

W3 / Q2: predictor feature ablations. We train the predictor with each input removed, and with the target model's last hidden state either added to or substituted for the drafter state. All variants use the same labels and training recipe with the retrained 4B drafter. The table reports the gain of plan-once over the best global depth in exact replay:

predictor inputsmixed workloadGSM8KMATH-500HumanEvalMBPPMT-Bench
full (drafter state + sorted probabilities + iteration)+2.1%+2.2%+0.6%+1.8%+1.8%+1.5%
without pooled drafter state+0.3%−0.4%−0.7%−1.4%+0.5%+1.0%
without sorted probabilities+1.8%+2.0%+0.6%+0.9%+0.9%+1.3%
without iteration embedding+1.3%+1.3%0.0%+1.2%+1.0%+1.2%
plus target hidden state+1.6%+2.2%+0.3%+1.1%+0.9%+1.1%
target hidden state instead of drafter state+0.2%+2.5%−1.4%−1.4%−0.9%+0.1%

The pooled drafter state is the most important input in this ablation. Removing it, or replacing it with the target hidden state, reduces the mixed-workload gain from 2.1% to 0.2–0.3%. Sorted per-position probabilities and the iteration embedding have smaller effects. Adding the target hidden state does not improve the predictor, so the deployed policy does not require additional target-model internals.

We observe the same pattern at 14B: removing the drafter state reduces the mixed-workload gain from 1.7% to 0.6%, using only the target hidden state gives 1.0%, and the other variants remain within 0.4% of the full predictor. The pooled drafter state is therefore the primary source of predictive signal, and additional target-model information adds little. We train the predictor on states from several recurrent iterations and use the iteration embedding to indicate which iteration produced each state. At inference, however, we read only the iteration-1 state. The deployed policy therefore makes one early prediction instead of repeatedly deciding whether to continue.

AuthorsOpenReview ↗

Gentle reminder: author-reviewer discussion period ending in a few days

Dear Reviewer aATD,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we reorganized Section 5 so that each experiment now appears under its own heading, with the main comparisons consolidated in Table 3. We also compared ReDraft directly with the five-layer DFlash drafter. Under matched training data and recipe, the recurrent core is 1.5–6.7% faster across the five benchmarks at similar drafting latency, and the advantage comes from higher acceptance, not cheaper drafting. We also ran the predictor feature ablations you requested. The pooled drafter state provides the main predictive signal, while adding the target hidden state does not improve the predictor.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

Reviewer_rKKYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReDraft, an adaptive-depth approach for recurrent speculative decoding. The key observation is that different speculative blocks benefit from different amounts of drafter computation. ReDraft uses a weight-shared recurrent drafter, predicts the acceptance curve for candidate depths using verifier-derived supervision, and selects a depth using a hardware-aware throughput objective. Since the target verification procedure is unchanged, the method preserves the lossless property of speculative decoding.

Critical strengths
  • The paper identifies a real inefficiency in fixed-depth drafting. The oracle results show substantial block-level variation in the useful drafter depth.
  • The recurrent drafter is technically interesting. The training strategy makes multiple recurrent depths usable while requiring substantially fewer parameters than the corresponding distinct-layer drafter.
  • Predicting target acceptance is better aligned with speculative-decoding throughput than relying only on internal signals such as confidence or convergence. The experiments provide useful evidence that standard convergence-based exits transfer poorly across tasks and models.
  • The plan-once controller is practical. It adds limited scheduling overhead and avoids repeatedly interrupting drafting with intermediate decisions.
  • The experimental evaluation is broad, covering multiple benchmarks, target sizes, model families, temperatures, training scales, and batching conditions. The appendices also address several important implementation and evaluation details.
Critical weaknesses
  • The main concern is that the adaptive policy does not consistently outperform a simple fixed-depth baseline. For Qwen3-4B, plan-once is roughly tied with fixed depth (r=2) on several benchmarks, but performs worse on MBPP and noticeably worse on MT-Bench. For Qwen3-8B at temperature 0, plan-once is slower than fixed (r=2) on all five reported benchmarks.
  • The paper convincingly demonstrates that useful drafter depth varies across blocks and that ReDraft approaches the best depth selected separately for each task. However, it does not yet convincingly demonstrate that adaptive depth selection is better than using one reasonable shallow depth across workloads.
  • The reported 4.40x speedup appears to validate the recurrent drafter and its training procedure more directly than the adaptive controller. The paper should more clearly separate the improvement from recurrent drafting from the additional benefit of adaptive depth selection.
  • The serving evaluation is based mainly on one hardware configuration and replayed batching. Evaluation under realistic continuous batching or mixed workloads would strengthen the practical claims.
  • The parameter and training-data comparisons should be stated more precisely, particularly the distinction between parameters in the recurrent transformer core and total drafter parameters.
Questions for authors

How does plan-once compare with a single global (r=2) policy on a mixed workload containing math, code, and dialogue requests?

Can the authors separate the speedup obtained from the recurrent drafter from the additional benefit of adaptive depth selection?

What is the computational cost of harvesting acceptance labels for all candidate depths, and how often must those labels be regenerated after changing the target model, temperature, domain, or hardware?

Does the benefit remain under real continuous batching rather than replayed synchronized batches?

Does “40% of the parameters” refer to the recurrent transformer core or to all drafter and policy parameters?

6 comments
AuthorsOpenReview ↗

Response to Reviewer rKKY (1/3)

We thank the reviewer for the careful review and for identifying the key question: whether adaptive depth improves on a reasonable shallow global depth such as r=2r=2. We now evaluate this comparison directly on a mixed workload and under changing drafting costs.

W1 / Q1: plan-once compared with global r=2r=2 on a mixed workload. We retrain the Qwen3-4B drafter and predictor and evaluate them on GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench. In wall-clock decoding, plan-once reaches 91.1 tokens/s, compared with 91.2 for global r=2r=2 and 91.3 for global r=3r=3, the best global depth chosen in hindsight. Thus, at the measured operating point, plan-once is within 0.2% of the best global depth without a deployment-time depth sweep. We also replay each block using its harvested acceptance at each candidate depth. This isolates the allocation policy from sampling noise. In this replay, plan-once is 2.1% faster than global r=3r=3 and 4.4% faster than global r=2r=2, and it outperforms the best global depth on all five benchmarks by 0.6–2.2%. The replay does not charge the predictor overhead of about 0.6 ms per block, including the first-iteration readout, so these gains are larger than the corresponding end-to-end gains. On Qwen3-8B, plan-once trailed fixed r=2r=2 under the original protocol, but with a retrained drafter and predictor it comes within 0–1.9% of the best fixed depth on each benchmark. On the larger Qwen3-14B target, it comes within 0.9–1.5% of the best fixed depth on each benchmark (Section 5.4). The selected depths also show that plan-once does not collapse to a single global depth. On GSM8K, it selects r=1,2,3,4r=1,2,3,4 for 20%, 31%, 41%, and 8% of blocks, respectively, and on MT-Bench, 47% of blocks use r=1r=1. The policy also changes depth within individual generations. For comparison, the closed-loop value head is 13.5% slower than the best global depth, while the KL-convergence exit is 23.5% slower.

The advantage becomes clearer when drafting costs change. The best fixed depth depends on the ratio between drafting and verification cost, which can vary across hardware and serving conditions. Because we measure tdt_d and tvt_v at startup, hardware changes do not require relabeling. We replay the same workload while scaling tdt_d from 0.25 to 4 times its measured value. The second row reports plan-once relative to the best global depth, while the last two rows report each global depth relative to plan-once:

tdt_d / measured0.250.511.5234
best global depthr=4r=4r=3r=3r=3r=3r=3r=3r=2r=2r=2r=2r=2r=2
plan-once relative to best global+0.1%+0.8%+2.1%+3.7%+4.1%+5.0%+6.2%
global r=2r=2 relative to plan-once−5.9%−5.0%−4.2%−3.9%−4.0%−4.8%−5.8%
global r=3r=3 relative to plan-once−0.5%−0.8%−2.0%−3.6%−5.3%−8.9%−12.1%

In cost-scaled replay, plan-once matches or exceeds the throughput of the best global depth at each cost. A depth tuned at the measured operating point can become substantially suboptimal as drafting becomes more expensive: r=3r=3 loses up to 12.1% at four times the drafting cost, while the best global depth shifts from r=3r=3 to r=2r=2. Plan-once uses the same decision rule across these regimes, without another depth sweep. The revision adds this mixed-workload result, the depth distribution, and the cost-sensitivity analysis to Section 5.2.

AuthorsOpenReview ↗

Response to Reviewer rKKY (2/3)

Q2: separating the drafter's contribution from the policy's. We evaluate Qwen3-4B at temperature 0 on the same hardware and serving harness, using 200 prompts per benchmark and one run per row. Speedup is relative to autoregressive decoding, and acceptance length appears in parentheses:

comparisondrafter (training data)depthGSM8KMATH-500HumanEvalMBPPMT-Bench
released systemDFlash-5, 5 distinct layers (own data)fixed4.39× (6.08)5.36× (7.38)4.69× (6.43)4.19× (5.64)2.43× (3.20)
training datarecurrent 2-layer core (main corpus + 120K code prompts)best fixed per task3.75× (4.95)3.98× (5.40)3.18× (4.49)2.98× (3.95)1.98× (2.55)
policyrecurrent 2-layer core (main corpus)best fixed per task3.87× (5.34)4.01× (5.50)2.61× (3.74)2.65× (3.55)1.96× (2.52)
policyrecurrent 2-layer core (main corpus)plan-once3.81× (5.21)3.96× (5.37)2.62× (3.70)2.61× (3.58)1.97× (2.51)

These rows separate three effects:

  1. Drafter architecture. Under matched data and training recipe, the five-layer drafter has 1.5–6.7% lower mean throughput than the recurrent core across the five benchmarks (7–10 seeds). The difference is largest on MATH-500, while the seed ranges overlap on the other benchmarks. In this controlled comparison, weight sharing does not incur a throughput penalty.
  2. Depth policy. With the recurrent drafter fixed, a single plan-once policy without per-task tuning comes within 1.7% of the best fixed depth selected separately for each benchmark, and slightly exceeds it on HumanEval and MT-Bench.
  3. Training data. The released DFlash-5 uses a different and larger corpus, so its advantage outside GSM8K does not isolate architecture. Adding 120K code-instruction prompts raises HumanEval from 2.61× to 3.18× and MBPP from 2.65× to 2.98×, while MATH-500 changes by about 1% and GSM8K by about 3%. Training data therefore accounts for much of the remaining code gap.

Under the paper's original protocol (128 prompts, three seeds), the recurrent drafter trained on the full 400K-sequence corpus reaches 4.40× on GSM8K, compared with 4.39× for the released DFlash-5. The recurrent core uses 40% as many distinct transformer-layer parameters and half as many training sample-epochs. We report these two evaluation protocols separately in Section 5.4 so that the controlled architectural comparison is not conflated with the released-checkpoint comparison.

Q3: label-harvesting cost and relabeling conditions. Harvesting labels for all candidate depths takes under one GPU-hour for 600 prompts and yields about 600K labeled examples, and predictor training takes minutes. Expanding the candidate grid from {1,2,3,4,6,8}\{1,2,3,4,6,8\} to {1,…,8}\{1,\ldots,8\} adds only 0.014 accepted tokens per block to the oracle, and the oracle on the finer grid selects depth 5 or 7 for only 4.5% of blocks. We relabel when the drafter changes: reusing a predictor for another 100K training steps increases its acceptance-curve error from 1.04 to 1.25 tokens. We also relabel when the serving temperature changes, and temperature-matched labels reduce the GSM8K gap to 1.9%. For a new target model, we regenerate labels for the corresponding drafter-target pair. Hardware changes do not require relabeling because they affect the policy only through the measured costs tdt_d and tvt_v. Across a ±20% grid of both costs, the selected depth ordering is unchanged in 8 of 9 settings. Domain transfer is more nuanced. One predictor trained on GSM8K plus held-out training-corpus prompts transfers to all five Qwen3 benchmarks, but on Llama, adding a 300-prompt MATH-500 harvest reduces the MATH-500 gap from 7.6% to 5.4%. Thus, a new domain does not necessarily require full retraining, but a small additional harvest can recover performance when the acceptance distribution shifts. Section 3.2 now summarizes these relabeling conditions.

AuthorsOpenReview ↗

Response to Reviewer rKKY (3/3)

Q4: continuous batching. We evaluate batching in two ways. First, in the static-batch replay of Appendix F, sorting blocks by planned depth reduces the straggler gap at B=16B=16 from 6.7% to 0.1% on GSM8K. Second, we build a continuous-batching simulator using measured costs and real acceptance traces. Requests from the mixed workload arrive as a Poisson process, their lengths come from the wall-clock runs, and up to BB requests remain active at once. Each engine step drafts one block for each active request, performs batched target verification, retires finished requests, and admits waiting requests with batched prefill. For the per-block policy, each request chooses its own depth before drafting. Because requests in a batch can choose different depths, the batch is drafted to the deepest planned depth, so requests that planned a shallower depth also pay for and receive the deeper draft. For the batch-level variant, the predicted curves instead choose one depth for the entire batch by maximizing predicted batch throughput. We measure drafting, verification, and prefill costs on our hardware for B=1B=1–32, charge the predictor overhead, and obtain acceptance at each candidate depth from the harvested labels. The resulting full-load throughput, in tokens/s, is:

active requests BB1481632
best fixed depth (r=3r=3)77.4301.1584.11095.31455.4
plan-once, per-block depths78.1296.8569.71063.51423.9
batch-level depth from predicted curves77.8298.8577.91082.01442.8
oracle batch depth (from true curves)90.4328.0616.41131.11482.0

Because the simulator drafts all active requests to a single depth, choosing that depth from the batch's predicted curves is the natural plan-once variant here, and it stays within 1.2% of the best fixed depth at each batch size. The per-block policy is slightly faster than the best fixed depth at B=1B=1 and within 2.9% at larger batch sizes, where drafting the whole batch to the deepest planned depth creates the remaining gap. At a load of 2 requests/s, the batch-level variant has a median time per output token of 14.6 ms, compared with 14.4 ms for the best fixed depth. Choosing the batch depth from the true acceptance curves would recover 2–17% over the best fixed depth, so request-level variation in acceptance still leaves substantial headroom under batching, even though our current predictor does not recover all of it. With the depth-sorting result above, these simulations suggest that the early depth decision does not introduce a large straggler penalty. We label these results explicitly as simulations, and production continuous batching remains future work.

Q5 / W: what "40% of the parameters" means. The 40% figure refers only to transformer-layer parameters: our recurrent core has two shared layers, compared with five distinct layers in DFlash. It does not mean 40% of total parameters. The complete recurrent drafter has 248M parameters, compared with 537M for the matched five-layer drafter (46%), and the acceptance-curve predictor adds 5M parameters. Similarly, "half the training data" refers to training sample-epochs in the original comparison. The revision reports layer-parameter and total-parameter figures separately and limits the serving claim to the evaluated single-request path, replayed batches, and measured-cost batching simulation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank the reviewers for the careful and constructive feedback. We have uploaded a revised version of the paper, with all changes highlighted in blue. Section, table, and figure numbers refer to the revised version. Across the reviews, the central question is whether adaptive depth adds value beyond a well-chosen fixed depth. We address this question with four sets of results and changes:

  1. Mixed workload. We retrain the Qwen3-4B drafter and predictor and evaluate them on five benchmarks. Plan-once reaches 91.1 tokens/s in wall-clock decoding, compared with 91.3 tokens/s for the best global depth chosen in hindsight, without a deployment-time depth sweep. In cost-scaled replay, the best global depth changes as drafting cost changes, while plan-once matches or exceeds its throughput at each cost.
  2. Drafter and policy effects. Under matched training, the five-layer drafter has 1.5–6.7% lower mean throughput than our recurrent core, while a single untuned plan-once policy comes within 1.7% of the best fixed depth on each benchmark. Adding 120K code prompts closes much of the remaining gap to the released DFlash-5 checkpoint on the code benchmarks.
  3. Larger targets and batching. With retrained drafters and predictors, plan-once comes within 0–1.9% of the best fixed depth on each benchmark at Qwen3-8B and within 0.9–1.5% at Qwen3-14B. In our continuous-batching simulation with measured costs, the batch-level plan-once variant stays within 1.2% of the best fixed depth from 1 to 32 active requests. We also initialize our recurrent core from the DSpark backbone and find that depth allocation remains effective. DSpark adapts how many drafted tokens to verify, whereas ReDraft adapts how much computation the drafter spends on each block.
  4. Predictor and presentation. We add a predictor-input ablation, clarify the parameter and training-data comparisons, consolidate the main comparisons in Section 5, and redraw Figure 2 to make the execution flow and indexing unambiguous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.