acceptodds
Under review as a conference paper at ICLR 2027

DLoop: Looped Speculative Decoding

13 views

Abstract

Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding.

Primary area:
generative models
Keywords:
Speculative decoding, large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LgquOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DLoop lets one target-model verification cover multiple drafting stages. A log-probability gate decides after each stage whether to keep drafting, and later stages condition on the drafter’s own hidden states for unverified tokens. A one-epoch “loop-aware” fine-tune teaches released drafters to use these states. Across EAGLE-3, DFlash, Domino, DSpark, and MTP modules, the paper reports 5–41% speedup gains under greedy decoding while remaining lossless.

Critical strengths
  1. The method is simple, lossless, and introduces no additional parameters. Unlike existing adaptive draft length methods, it also benefits parallel drafters.
  2. The evaluation covers a wide range of drafters and target models. The analysis in Fig. 3, Table 4, and Table A4 clearly identifies the source of the speedup.
  3. The ablation in Table 2 shows that loop-aware training and the drafting loop are complementary. Training alone has little effect and the loop alone mainly benefits autoregressive drafters, while their combination improves all settings.
Critical weaknesses
  1. The proposed method only works under low concurrencies and is not practical for large-batch serving. In real-world serving, the draft block size is usually 4 or 8 because of the non-negligible verification cost. In this setting, doubling the verification length would not benefit serving throughput.
  2. No comparison with drafters trained for a larger block size. DLoop verifies up to S·k tokens per verification. The obvious alternative is to fine-tune the same drafter with block size 2k or 3k, which would be compute-comparable to the one-epoch loop-aware training. Especially, for parallel drafters, DLoop’s drafting cost grows nearly linearly with S (Table A4: DFlash 3.66 → 13.96 ms for S=1–4), whereas a 2k block takes a single forward pass. Without this baseline, it is unclear whether the gains come from looping or simply from drafting more tokens with a drafter trained for that length.
  3. The motivation and headline results are greedy-only. Fig. 1 is measured at T=0. At T=1 (Table A10), gains shrink (DSpark on Qwen3-4B: +26.3% → +16.3%), and Domino on Qwen3-8B gets slightly slower (3.06× → 3.04×). The abstract should state the decoding setting, and the paper should report full acceptance at T=1.
Questions for authors
  1. How does DLoop compare with drafters fine-tuned for 2k or 3k block sizes, or with a larger draft depth for EAGLE-3 and MTP?
  2. How does a fixed S=2 or S=3 perform without the gate?
  3. What is the full acceptance ratio at T=1?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Lgqu (Part1)

Table, figure, section, and appendix numbers of the paper in our responses refer to the revised manuscript.

W1: Practicality in large-batch serving

Thank you for raising this practical concern. Limiting the loop depth of each request recovers the gain of DLoop at high concurrency, and DLoop then improves throughput over the draft model at every concurrency up to c=32c = 32, including DSpark with k=7k = 7 and MTP with k=3k = 3. As discussed in Section 4.2, DLoop chooses the number of drafting stages independently for each request, which leads to different loop depths within a batch. We agree that these different depths, together with the additional speculative work on a saturated target model, reduce the gain of the loop in Table 6 at high concurrency. However, simply capping the loop depth of each request at a maximum Smax⁡S_{\max} reduces both effects.

Our new experiment 1. Bounded loop depth. We cap the loop depth at Smax⁡=3S_{\max} = 3 and measure DFlash and DSpark in SGLang under the same setting.

Table R1. Results in SGLang with Qwen3-8B (throughput in tokens(tok)/s and speedup over autoregressive decoding). Bold marks results above the draft model.

Methodc=1c = 1c=2c = 2c=4c = 4c=8c = 8c=16c = 16c=32c = 32
MATH-500
DFlash907 / 5.81×1,750 / 5.78×3,252 / 5.48×6,019 / 5.29×9,484 / 4.45×11,810 / 3.14×
+ DLoop972 / 6.23×1,833 / 6.05×3,362 / 5.67×6,134 / 5.39×9,189 / 4.31×11,139 / 2.96×
+ DLoop (Smax⁡=3S_{\max}{=}3)933 / 5.98×1,792 / 5.91×3,361 / 5.67×6,129 / 5.39×9,571 / 4.49×12,015 / 3.19×
DSpark694 / 4.45×1,244 / 4.11×2,517 / 4.24×4,538 / 3.99×8,117 / 3.81×12,265 / 3.26×
+ DLoop770 / 4.94×1,480 / 4.88×2,692 / 4.54×4,874 / 4.28×8,266 / 3.88×12,820 / 3.41×
+ DLoop (Smax⁡=3S_{\max}{=}3)742 / 4.76×1,321 / 4.36×2,573 / 4.34×4,803 / 4.22×8,294 / 3.89×13,020 / 3.46×
HumanEval
DFlash717 / 4.60×1,357 / 4.51×2,552 / 4.29×4,673 / 4.07×7,489 / 3.48×9,451 / 2.51×
+ DLoop779 / 4.98×1,454 / 4.83×2,648 / 4.45×4,791 / 4.17×7,668 / 3.56×9,104 / 2.42×
+ DLoop (Smax⁡=3S_{\max}{=}3)737 / 4.72×1,446 / 4.80×2,636 / 4.43×4,841 / 4.21×7,711 / 3.58×9,545 / 2.54×
DSpark641 / 4.11×1,228 / 4.07×2,243 / 3.77×4,232 / 3.68×7,234 / 3.36×10,926 / 2.90×
+ DLoop714 / 4.58×1,326 / 4.41×2,451 / 4.12×4,317 / 3.76×7,297 / 3.39×10,825 / 2.88×
+ DLoop (Smax⁡=3S_{\max}{=}3)682 / 4.37×1,280 / 4.25×2,459 / 4.13×4,329 / 3.77×7,471 / 3.47×11,105 / 2.95×

With Smax⁡=3S_{\max} = 3, DLoop exceeds the draft model at every concurrency for both draft models and both benchmarks. At high load, a smaller Smax⁡S_{\max} keeps the loop depths within a batch close and limits the speculative work on the saturated target model.

Our new experiment 2. To cover the small block sizes that the reviewer mentions, we also measure DLoop in SGLang with the MTP module of Qwen3.5-9B, which drafts k=3k = 3 tokens per drafting stage. With Smax⁡=3S_{\max} = 3, one verification covers at most 99 draft tokens.

Table R2. Results in SGLang with the MTP module of Qwen3.5-9B (k=3k = 3). The notation follows Table R1.

Methodc=1c = 1c=2c = 2c=4c = 4c=8c = 8c=16c = 16c=32c = 32
MATH-500
MTP353 / 2.28×676 / 2.24×1,280 / 2.18×2,293 / 2.09×3,866 / 1.93×5,920 / 1.72×
+ DLoop407 / 2.63×717 / 2.37×1,389 / 2.37×2,341 / 2.13×3,648 / 1.82×5,545 / 1.61×
+ DLoop (Smax⁡=3S_{\max}{=}3)392 / 2.53×761 / 2.52×1,432 / 2.44×2,556 / 2.33×4,186 / 2.09×6,227 / 1.81×
HumanEval
MTP331 / 2.15×626 / 2.10×1,153 / 2.02×1,956 / 1.91×3,146 / 1.74×4,383 / 1.54×
+ DLoop372 / 2.42×639 / 2.14×1,259 / 2.20×2,070 / 2.02×3,288 / 1.82×4,576 / 1.61×
+ DLoop (Smax⁡=3S_{\max}{=}3)356 / 2.31×688 / 2.31×1,285 / 2.25×2,150 / 2.10×3,272 / 1.81×4,582 / 1.61×

With Smax⁡=3S_{\max} = 3, DLoop exceeds MTP at every concurrency on both benchmarks. At c=32c = 32, it raises the throughput from 5,9205,920 to 6,2276,227 tok/s on MATH-500 and from 4,3834,383 to 4,5824,582 tok/s on HumanEval. These results suggest that adjusting the loop depth of each request with the load keeps DLoop beneficial in large-batch serving with small draft blocks, in line with the shorter optimal speculation length at high load (Su et al., 2023; Liu et al., 2024b; Li et al., 2025; Cheng et al., 2026).

We have added this discussion and the new results to Section 4.2 of the main paper (Table 6).

AuthorsOpenReview ↗

Response to Reviewer Lgqu (Part2)

W2 & Q1: Comparison with drafters trained for larger blocks or deeper drafts

Thank you for suggesting this baseline. The new results indicate that the gain of DLoop does not come simply from drafting more tokens, since drafters fine-tuned for 2k2k and 3k3k blocks stay below kk + DLoop in speedup, and DLoop further improves both of them.

Table R3. Acceptance length and speedup (τ\tau / spd×) of DSpark trained with block sizes kk, 2k2k, and 3k3k, with and without DLoop, using Qwen3-8B at T=0T = 0.

SetupMathCodeChatAvg
kk6.32 / 4.46×5.81 / 3.96×4.13 / 2.61×5.58 / 3.81×
kk + DLoop11.80 / 6.20×8.84 / 4.83×5.39 / 2.74×9.08 / 4.82×
2k2k8.14 / 5.49×7.12 / 4.73×4.51 / 2.59×6.85 / 4.48×
2k2k + DLoop11.58 / 6.57×8.42 / 5.02×5.27 / 2.62×8.82 / 5.00×
3k3k9.28 / 5.76×7.52 / 4.74×4.82 / 2.54×7.51 / 4.57×
3k3k + DLoop11.80 / 6.78×8.72 / 5.15×5.06 / 2.57×8.96 / 5.12×

Training for a larger block does not replace the loop. For Table R3, we fine-tune the kk-block DSpark for one more epoch with a 2k2k or 3k3k block, following the training recipe released with DSpark, as in loop-aware training, so that the training compute matches that of DLoop. Training for a larger block improves both the acceptance length and the speedup over the kk-block drafter on average. We agree that a 2k2k or 3k3k block is drafted in a single forward pass, whereas each DLoop stage adds a drafter pass (Table A6). However, neither the 2k2k nor the 3k3k drafter reaches the speedup of kk + DLoop. A larger block adds positions that are predicted farther from the verified context, and these positions are accepted less often, so the acceptance length grows much less than the number of drafted tokens.

Applying DLoop on top of the 2k2k and 3k3k drafters further improves the speedup, which indicates that the gain of DLoop goes beyond drafting more tokens with a drafter trained for that length. The gate adds a drafting stage only after a confident block, so DLoop verifies more tokens per target forward pass without spending drafting stages on blocks that are later discarded.

Table R4. Acceptance length and speedup (τ\tau / spd×) of EAGLE-3 with Qwen3-8B and MTP with Qwen3.5-9B for different draft depths at T=0T = 0. "+ Loop only" enables the confidence-gated drafting loop, and "+ DLoop" also applies loop-aware training. Bold marks the best speedup in each column for each draft model.

MethodMathCodeChatAvg.
EAGLE-3 (Qwen3-8B)
Depth 7 (default)7.59 / 3.78×6.73 / 3.30×4.98 / 2.26×6.61 / 3.22×
Depth 108.78 / 3.88×7.32 / 3.20×5.22 / 1.99×7.34 / 3.15×
Depth 149.23 / 3.54×7.38 / 2.83×5.30 / 1.75×7.56 / 2.82×
Depth 7 + Loop only8.97 / 4.10×7.16 / 3.37×5.14 / 2.33×7.34 / 3.38×
Depth 14 + Loop only10.88 / 3.82×8.05 / 3.07×5.59 / 1.94×8.50 / 3.07×
Depth 7 + DLoop10.45 / 4.31×7.83 / 3.46×5.46 / 2.38×8.22 / 3.51×
MTP (Qwen3.5-9B)
Depth 3 (default)3.54 / 1.74×3.39 / 1.58×2.88 / 1.21×3.32 / 1.55×
Depth 44.15 / 1.90×3.93 / 1.72×3.18 / 1.27×3.82 / 1.68×
Depth 54.64 / 1.97×4.35 / 1.79×3.41 / 1.25×4.23 / 1.72×
Depth 65.05 / 2.02×4.69 / 1.80×3.56 / 1.22×4.54 / 1.74×
Depth 3 + Loop only5.66 / 2.10×5.17 / 1.86×3.57 / 1.27×4.96 / 1.80×
Depth 6 + Loop only5.90 / 2.07×5.50 / 1.87×3.83 / 1.24×5.23 / 1.79×
Depth 3 + DLoop8.26 / 2.66×6.92 / 2.26×4.02 / 1.33×6.70 / 2.18×

A larger draft depth shows the same trend. Table R4 increases the draft depth of EAGLE-3 and MTP and applies the loop at both the default and a larger depth. A larger depth raises the acceptance length, but the speedup does not rise accordingly, since the additional drafting steps are spent in every drafting stage, including those whose later draft tokens are rejected. The gate instead adds a drafting stage only after a confident block, and DLoop at the default depth gives the highest speedup for both draft models. Even on top of a larger depth and without loop-aware training, the loop still raises the speedup, which indicates that the gain comes from looping rather than from drafting more tokens.

We have added Tables R3 and R4 and this discussion to Appendix A.10.1.

AuthorsOpenReview ↗

Response to Reviewer Lgqu (Part3)

Q2: Fixed number of drafting stages versus the gate

Thank you for this question. A fixed SS raises the acceptance length but lowers the speedup in most settings, whereas the gate gives the highest speedup for every draft model in Table R5.

Table R5. Fixed SS versus the gate. All rows use the same loop-aware trained drafter and differ only in how SS is set. Average τ\tau / speedup over the eight benchmarks on Qwen3-8B at T=0T = 0.

MethodEAGLE-3DFlashDominoDSpark
S=1S = 1 (no loop)6.60 / 3.15×6.48 / 4.33×7.23 / 4.52×5.50 / 3.77×
S=2S = 2 (fixed)8.89 / 3.21×7.30 / 4.14×8.25 / 4.07×7.45 / 4.25×
S=3S = 3 (fixed)9.72 / 2.77×7.53 / 3.75×8.55 / 3.54×8.45 / 4.20×
DLoop (gate)8.22 / 3.51×7.40 / 4.71×8.29 / 4.76×9.08 / 4.82×

With a fixed SS, every loop performs the additional drafting stages even after an uncertain block, and a rejection in an earlier block discards all later blocks. The acceptance length therefore increases with SS, but the cost of the additional drafting stages outweighs this gain, and the speedup falls below that of S=1S = 1 already at S=2S = 2 for DFlash and Domino and at S=3S = 3 for EAGLE-3. DSpark gains from a fixed SS, but its gain stays well below that of the gate. The gate instead adds a drafting stage only after a confident block, which avoids the drafting stages that a fixed SS spends on blocks that are later discarded.

We have added Table R5 and this discussion to Appendix A.10.2.


W3 & Q3: Results at T=1 and full acceptance ratio

Thank you for pointing this out. The full acceptance ratio at T=1T = 1 is lower than at T=0T = 0 for every draft model in Table R6, both with and without tree verification, consistent with the lower acceptance length of the draft models themselves at T=1T = 1 in Table A17.

Table R6. Full acceptance ratio (%) at T=0T = 0 / T=1T = 1, averaged over the benchmarks in each task category and over all eight benchmarks (Avg.). Tree verification of EAGLE-3 is part of its original recipe.

Draft modelMathCodeChatAvg.
Without tree verification
EAGLE-3 (Qwen3-4B)39.4 / 26.728.2 / 21.08.1 / 5.827.4 / 19.3
DFlash (Qwen3-4B)13.6 / 7.913.4 / 7.31.2 / 1.110.4 / 6.0
Domino (Qwen3-4B)19.6 / 8.912.6 / 4.81.8 / 0.812.5 / 5.3
DSpark (Qwen3-4B)55.8 / 45.944.3 / 36.215.6 / 12.941.4 / 34.0
EAGLE-3 (Qwen3-8B)32.4 / 22.820.4 / 13.95.7 / 3.921.2 / 14.8
DFlash (Qwen3-8B)14.6 / 8.312.4 / 7.81.1 / 0.810.4 / 6.3
Domino (Qwen3-8B)21.6 / 9.212.3 / 5.12.3 / 0.913.3 / 5.6
DSpark (Qwen3-8B)57.5 / 46.146.6 / 39.116.0 / 13.043.0 / 35.2
With tree verification
EAGLE-3 (Qwen3-4B)68.0 / 51.952.6 / 44.818.3 / 14.449.8 / 39.8
DFlash + DDTree (Qwen3-4B)23.2 / 15.722.1 / 16.42.8 / 2.317.7 / 12.6
Domino + DARTree (Qwen3-4B)31.8 / 19.720.9 / 12.83.1 / 2.320.5 / 12.8
DSpark + DARTree (Qwen3-4B)67.3 / 52.762.5 / 51.125.1 / 19.654.9 / 43.8
EAGLE-3 (Qwen3-8B)61.2 / 46.144.8 / 34.513.4 / 10.343.1 / 32.8
DFlash + DDTree (Qwen3-8B)23.5 / 15.422.5 / 13.92.5 / 1.817.8 / 11.5
Domino + DARTree (Qwen3-8B)35.2 / 20.021.9 / 13.63.3 / 2.322.3 / 13.2
DSpark + DARTree (Qwen3-8B)71.6 / 56.964.9 / 54.528.3 / 21.758.3 / 47.2

A lower full acceptance ratio leaves fewer blocks after which an additional drafting stage can be accepted, which is consistent with the smaller gain at T=1T = 1. Without tree verification, Domino has the lowest full acceptance ratio at T=1T = 1 for both target models. However, tree verification raises the full acceptance ratio of every draft model in Table R6. With tree verification, DLoop raises the speedup at T=1T = 1 for every draft model in Table A17, including Domino on Qwen3-8B from 4.16×4.16× to 4.36×4.36×.

We have stated the decoding setting in the abstract and added Table R6 to Appendix A.3.

Reviewer_1dkfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes looped drafting stage for speculative decoding that continues to draft tokens as long as the cumulative log probability of individual tokens within a drafting window is over a pre-defined threshold for every loop.

Critical strengths

The idea is technically sound, even if not entirely novel. Results demonstrate the effectiveness of this approach.

Overall, the paper is well written and shows meaningful improvement. I recommend acceptance for this paper.

Critical weaknesses

One question I have is for the decision of this threshold. It would be more meaningful if ‘g’ could be made adaptive based on application types. Currently, the authors have selected the value of ‘g’ as -0.5 for most benchmarks except 1 and this seems to be done manually through a sweep from -0.25 to -1. Is there a way to automate this decision?

Considering the example in Figure 2, could you also take into the account the runtime of drafting vs verification stage to determine a roofline for the number of loops before the drafting time overtakes the verification time? In practice, this may not happen often, but this may serve as a good way to determine the upper bound for the number of times the loop runs to make it balance with the verification stage. A well balanced drafting and verification can run well in a pipeline mode.

Questions for authors

See above

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1dkf

Table, figure, section, and appendix numbers of the paper in our responses refer to the revised manuscript.

Q1: Automatic or application-adaptive choice of g

Thank you for this suggestion. The threshold gg can be set automatically during decoding, and the gg found this way converges close to the best fixed gg of each domain, with an average speedup within 1.11.1% of our swept gg.

Our new experiment. We adjust gg during decoding from the verification outcomes with the Robbins–Monro update [1], g←g−η(y−ρ)g \leftarrow g - \eta (y - \rho). Here y=1y = 1 if a block that the gate let continue is fully accepted and y=0y = 0 otherwise, ρ=0.9\rho = 0.9 is the target fraction of such blocks that are fully accepted, and η=0.1\eta = 0.1 is the step size. The rule starts from g=−1.0g = -1.0 at every prompt, needs no sweep of gg, and adds no forward pass or trained module. We also report the best fixed gg chosen separately for each domain.

Table R1. DSpark with Qwen3-4B at T=0T = 0. Each cell reports average acceptance length ττ / speedup over the eight benchmarks.

MethodMathCodeChatAvg
Fixed gg11.02 / 6.02×8.44 / 4.86×5.17 / 2.71×8.59 / 4.76×
Fixed gg, best per domain11.02 / 6.02×8.44 / 4.86×4.98 / 2.76×8.54 / 4.77×
Adaptive gg (ρρ 0.9, start at −1.0)10.87 / 5.86×7.71 / 4.88×5.06 / 2.74×8.23 / 4.71×

Table R2. Converged adaptive gg (mean over the second half of decoding) and the best fixed gg of each domain.

MathCodeChat
Adaptive gg−0.58−0.46−0.29
Best fixed gg per domain−0.5−0.5−0.25

Choosing gg separately for each domain raises the average speedup only from 4.76×4.76\times to 4.77×4.77\times, which indicates little gain from tuning gg per application. Adaptive gg reaches 4.71×4.71\times without any sweep, slightly above the fixed gg on code (4.88×4.88\times vs. 4.86×4.86\times) and chat (2.74×2.74\times vs. 2.71×2.71\times) but below it on math (5.86×5.86\times vs. 6.02×6.02\times). Interestingly, the converged gg lands close to the best fixed gg of each domain in Table R2, at −0.58-0.58, −0.46-0.46, and −0.29-0.29 against −0.5-0.5, −0.5-0.5, and −0.25-0.25. These results suggest that gg can be set automatically at a small cost in speedup.

We have added both tables and the adaptive rule to Appendix A.9.4.

[1] Robbins, H. and Monro, S., 1951. A stochastic approximation method. The annals of mathematical statistics, pp.400-407.


Q2: Runtime-based upper bound on the loop depth

Thank you for this suggestion. The runtime-based bound from Fig. 2 gives Smax⁡=5S_{\max} = 5, and the speedup of DLoop saturates at this bound in Table R3. In Fig. 2, one drafting stage takes 3.903.90 ms and one verification takes 22.4522.45 ms, so the total drafting time stays below one verification time up to five drafting stages (19.5019.50 ms) and exceeds it at six (23.4023.40 ms). We measure DLoop with this bound and with other values of Smax⁡S_{\max}, where Smax⁡=∞S_{\max} = \infty is DLoop in Table 1.

Table R3. Average acceptance length and speedup (τ\tau / spd×) over the eight benchmarks by Smax⁡S_{\max}, with DSpark and Qwen3-4B at T=0T = 0. †^\dagger marks the runtime-based bound from Fig. 2. Bold marks the highest value of each metric.

Smax⁡S_{\max}1 (no loop)2345$^\dagger$678∞\infty
DSpark (Qwen3-4B)5.41 / 3.73×6.73 / 4.27×7.58 / 4.67×7.94 / 4.75×8.11 / 4.77×8.21 / 4.76×8.28 / 4.77×8.35 / 4.77×8.59 / 4.76×

The speedup increases up to the bound and changes little beyond it, although the acceptance length keeps increasing. Beyond the bound, the drafting time of a loop exceeds one verification time, which is consistent with the added drafting time offsetting the gain in acceptance length. The runtime-based bound therefore balances drafting and verification within each loop without reducing the speedup, as the reviewer suggests.

We have added Table R3 and the runtime-based bound to Appendix A.9.5.

AuthorsOpenReview ↗

Follow-up to Reviewer 1dkf

Dear Reviewer 1dkf,

Thank you again for your thoughtful review and for your positive assessment of our work.

As the discussion deadline is approaching, we would appreciate it if you could let us know whether our responses have addressed your questions.

If you have any further questions or suggestions, please let us know and we will be happy to clarify.

Best regards,

The Authors

AuthorsOpenReview ↗

Gentle Reminder on Our Rebuttal

Dear Reviewer 1dkf,

We understand that this is a busy period, and we sincerely appreciate your time and positive assessment. We would like to briefly follow up on our rebuttal, which provides the following.

  • An additional experiment to address Q1 (an adaptive rule that sets g automatically during decoding)
  • An additional experiment to address Q2 (a runtime-based upper bound on the loop depth)

Both are reflected in the revised manuscript. We would greatly appreciate it if you could take a look at our responses before the discussion period closes. We would be happy to address any remaining questions or concerns.

Best regards,

The Authors

Reviewer_S7jEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DLoop is a looped speculative decoding method that eliminates redundant computation in the standard draft-verify cycle by allowing a single verification to cover multiple drafting stages. Its confidence gate skips verification when the draft model remains confident, and loop-aware training extends EAGLE-3's training-time test to multi-stage settings for reliable continued drafting. Across EAGLE-3, DFlash, Domino, DSpark, and MTP modules, DLoop achieves 5–41% wall-clock speedup, unifies coverage of both parallel and autoregressive drafters, composes orthogonally with tree verification, and requires no architectural changes or additional parameters.

Critical strengths

The confidence gate requires only a single summation and comparison over already-computed log probabilities, adding virtually zero inference cost. This makes it a drop-in enhancement that plugs directly into any publicly released draft model (EAGLE-3, DFlash, Domino, DSpark) without retraining or service framework changes. In practice, this non-invasive property is enormously valuable — teams can extract speedup without assuming any architectural migration risk.

Critical weaknesses
  1. Limited novelty

The cumulative log-probability thresholding mechanism is virtually identical to LLMCad (2023) and DDD (2024). Loop-aware Training is a straightforward extension of EAGLE-3's training-time test, unrolling across multiple stages instead of within one. The overall method can be characterized as a natural composition of "gated continuation + EAGLE-3 training trick → multi-stage setting." The paper's primary contribution lies in the breadth of its experimental coverage (5 draft models × 2 target models × tree verification combinations) rather than in conceptual originality.

  1. Inadequate evaluation on realistic deployment scenarios

Single hardware configuration: All experiments run on a single H100 80GB GPU, with no coverage of other consumer-grade GPUs, or chips with different compute/memory characteristics. Fixed short context length: Evaluation uses only a 1024-token input prefix. Long-context reliability (32K, 128K) — a common requirement in production — is entirely untested. Single-sequence decoding, no batch serving: Real-world inference serving requires dynamic batching, where different sequences in the same batch may have vastly different optimal loop depths, introducing non-trivial scheduling complexity. This is not discussed.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer S7jE (Part1)

Table, figure, section, and appendix numbers of the paper in our responses refer to the revised manuscript.

W1: Novelty over LLMCad, DDD, and the training-time test of EAGLE-3

Thank you for the detailed comparison with prior work.

Confidence-gated drafting loop versus LLMCad and DDD. Our contribution is not a new confidence metric, but a drafting loop in which a single verification covers multiple drafting stages, together with the training that keeps the draft model reliable in those stages. LLMCad and DDD decide at the token level when to stop a draft model that can already continue on its own. DLoop addresses how another full drafting stage can follow before verification, without target-model hidden states for the unverified draft tokens. In short, the key distinction is that LLMCad and DDD decide when to stop drafting, whereas DLoop enables drafting to continue beyond a single stage without verification.

This distinction matters particularly for parallel draft models, which produce all draft tokens of a drafting stage in a single forward pass. For these draft models, the previous methods leave little room for speedup for two reasons. First, stopping at a token leaves the drafting time unchanged and barely reduces the verification (22.2±0.322.2 ± 0.3 ms for 22 to 1616 draft tokens, Appendix A.11.3). Second, extending the draft beyond one drafting stage, whether token by token or block by block, requires the draft model to condition on its own hidden states for unverified tokens, which it is not trained for without loop-aware training.

As shown in Table R1, LLMCad and DDD provide no speedup improvement over the released DSpark model, whereas DLoop improves average speedup from 3.77×3.77× to 4.76×4.76×.

Table R1. Comparison with LLMCad and DDD on the parallel draft model DSpark (average speedup over the eight benchmarks).

DSpark with Qwen3-4BSpeedup
Released draft model3.77×
+ LLMCad (Table 3)3.56×
+ DDD (Table 3)3.77×
+ DLoop4.76×

The results in Table R1 are consistent with this analysis.

Loop-aware training versus the training-time test. Loop-aware training and the training-time test train different parts of drafting. The training-time test trains token-level drafting within a stage from the draft model's own features, which a single-pass parallel draft model does not need. Loop-aware training teaches the draft model to start a new drafting stage before verification from its own hidden states for unverified draft tokens (Eq. 3). Every draft model needs this in the loop, even autoregressive ones (EAGLE-3 and MTP). EAGLE-3, trained with the training-time test, rises from 3.35×3.35× to 3.52×3.52× with the loop alone and to 3.77×3.77× with loop-aware training, and MTP with Gemma-4-12B goes from 3.17×3.17× to 2.90×2.90× and then 3.68×3.68× (Table 2). In terms of training cost, a direct application of the training-time test, supervising every stage (mid-CE in Table A9), also raises the training memory and lowers the speedup.

Why DLoop is not a simple composition. Therefore, DLoop is not a simple composition of gated continuation and the training-time test. Multi-stage drafting introduces a distinct challenge. The additional drafting stages must condition on the draft model's own hidden states for unverified tokens, which neither gated continuation nor the training-time test addresses. DLoop enables this by taking these hidden states as input (Eq. 3) and by loop-aware training. Table 2 indicates that the two components work jointly. The gain of DLoop over the released draft model exceeds the sum of the gains from the loop alone and loop-aware training alone in every setting.

We have added this discussion to Sections 2, 3.3, and 4.2.

AuthorsOpenReview ↗

Response to Reviewer S7jE (Part2)

W2-1: Evaluation on other GPUs

Thank you for raising the hardware coverage.

Our new experiment on A100. Since consumer-grade GPUs are not available in our environment, we evaluate DLoop on an A100, which has about one third of the bf16 compute and 0.6×0.6× the memory bandwidth of the H100 used in the paper. All other settings follow Table 1.

Table R2. Results on H100 and A100 (average speedup over the eight benchmarks, shown as draft model → + DLoop).

TargetDraft modelH100A100
Qwen3-4BEAGLE-33.35× → 3.77×4.58× → 5.27×
DFlash4.19× → 4.67×5.11× → 5.68×
Domino4.38× → 4.58×5.00× → 5.27×
DSpark3.77× → 4.76×4.33× → 5.33×
Qwen3-8BEAGLE-33.22× → 3.51×4.33× → 4.80×
DFlash4.15× → 4.71×5.14× → 5.86×
Domino4.47× → 4.76×5.09× → 5.37×
DSpark3.81× → 4.82×4.47× → 5.48×
Qwen3.5-9BMTP1.55× → 2.18×1.68× → 2.31×
Gemma-4-12BMTP3.17× → 3.68×3.24× → 3.82×

On the A100, DLoop raises the speedup in every setting by 55% to 3838%, close to the 55% to 4141% on the H100. DLoop replaces target-model forward passes with draft-model forward passes, which pays off as long as one drafting stage costs less than one verification.

We have added the A100 results to Appendix A.7.


W2-2: Clarification of the 1,024-token prefix; long-context evaluation

Thank you for raising the evaluation context length. The 1,0241,024-token prefix is used only to illustrate the latency of a drafting stage and a verification in Figure 2 and Table A6. All other results, including Tables 1 to 6, use the prompts of the eight benchmarks. We agree that the paper did not cover long contexts. However, DLoop also raises the speedup at 32K32K and 128K128K in our new experiment below.

Our new experiment on long contexts. We build 3232K and 128128K versions of two long-context benchmarks. For GSM-∞ [1], we sample 128128 problems at each length whose prompts reach the target length, following the cap of 128128 samples in Appendix A.1. For LongSWE-Bench (LSB) [2], we select the samples in its 3232K and 128128K buckets whose prompts reach at least 7575 percent of the bucket length, which leaves 1616 samples at 3232K and 4141 at 128128K.

Table R3. Results on long contexts (ττ and spd denote the acceptance length and speedup). DLoop uses the same gate thresholds as in the paper.

MethodLSB 32KGSM-∞ 32KLSB 128KGSM-∞ 128K
ττ / spdττ / spdττ / spdττ / spd
EAGLE-3 (Qwen3-8B)2.97 / 1.22×6.54 / 2.89×2.55 / 1.10×6.37 / 1.39×
+ DLoop3.68 / 1.52×8.19 / 3.14×3.13 / 1.35×7.37 / 1.85×
DFlash (Qwen3-8B)3.34 / 1.94×5.78 / 3.65×1.86 / 1.19×2.73 / 1.92×
+ DLoop4.14 / 2.52×7.33 / 4.33×2.20 / 1.49×4.73 / 3.17×
Domino (Qwen3-8B)3.48 / 1.94×7.93 / 4.72×2.46 / 1.15×5.12 / 2.41×
+ DLoop3.55 / 2.12×9.39 / 5.13×2.72 / 1.37×6.05 / 2.85×
DSpark (Qwen3-8B)3.41 / 2.14×5.77 / 3.91×2.59 / 1.17×4.99 / 2.52×
+ DLoop4.83 / 2.78×11.45 / 5.24×2.95 / 1.35×6.76 / 2.97×
MTP (Qwen3.5-9B)3.48 / 1.68×3.61 / 1.82×3.38 / 1.14×3.61 / 1.05×
+ DLoop4.77 / 1.90×5.13 / 2.15×3.97 / 1.34×5.06 / 1.24×
MTP (Gemma-4-12B)4.92 / 2.93×5.38 / 3.34×4.26 / 2.61×5.44 / 3.38×
+ DLoop5.10 / 3.23×7.41 / 3.64×4.57 / 2.92×8.09 / 3.84×

DLoop raises both the acceptance length and the speedup for every draft model at both context lengths, using the same gate threshold as in the paper without retuning. This indicates that the gain of DLoop carries over to long contexts.

We have added this clarification to Section 4.1 and the long-context results to Appendix A.6.

[1] Zhou, Y., Liu, H., Chen, Z., Tian, Y. and Chen, B., 2025. GSM-∞: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?. In ICML.
[2] Rando, S., Romani, L., Sampieri, A., Franco, L., Yang, J., Kyuragi, Y., Galasso, F. and Hashimoto, T., 2025, October. LongCodeBench: Evaluating Coding LLMs at 1M Context Windows. In COLM.

AuthorsOpenReview ↗

Response to Reviewer S7jE (Part3)

W2-3: Batch serving with per-request loop depths

Thank you for raising batch serving. Section 4.2 and Table 6 (Appendix A.3 and Table A2 in the original submission) already measure DLoop in SGLang, a widely used LLM serving framework, under continuous batching, where cc denotes the number of requests served concurrently. As discussed there, DLoop chooses the number of drafting stages independently for each request, and the resulting ragged draft-side execution requires either padding or dynamic regrouping. We agree that this adds scheduling complexity, which is consistent with the smaller gain of DLoop at high concurrency in Table 6. However, limiting the loop depth SS of each request to a maximum Smax⁡S_{\max} reduces this complexity. We report the results with Smax⁡=9S_{\max} = 9 and Smax⁡=3S_{\max} = 3 below, which allow at most 88 and 22 additional drafting stages, respectively, under the same setting of Table 6.

Table R4. Batch serving results in SGLang with Qwen3-8B (throughput (tokens per second) in tok/s and speedup over autoregressive decoding). Bold marks results above the draft model.

Methodc=1c = 1c=2c = 2c=4c = 4c=8c = 8c=16c = 16c=32c = 32
MATH-500
DFlash907 / 5.81×1,750 / 5.78×3,252 / 5.48×6,019 / 5.29×9,484 / 4.45×11,810 / 3.14×
+ DLoop972 / 6.23×1,833 / 6.05×3,362 / 5.67×6,134 / 5.39×9,189 / 4.31×11,139 / 2.96×
+ DLoop (Smax⁡=9S_{\max}{=}9)972 / 6.23×1,790 / 5.91×3,341 / 5.63×6,191 / 5.44×9,240 / 4.33×11,739 / 3.12×
+ DLoop (Smax⁡=3S_{\max}{=}3)933 / 5.98×1,792 / 5.91×3,361 / 5.67×6,129 / 5.39×9,571 / 4.49×12,015 / 3.19×
DSpark694 / 4.45×1,244 / 4.11×2,517 / 4.24×4,538 / 3.99×8,117 / 3.81×12,265 / 3.26×
+ DLoop770 / 4.94×1,480 / 4.88×2,692 / 4.54×4,874 / 4.28×8,266 / 3.88×12,820 / 3.41×
+ DLoop (Smax⁡=9S_{\max}{=}9)760 / 4.88×1,475 / 4.87×2,646 / 4.46×4,893 / 4.30×8,359 / 3.92×12,852 / 3.42×
+ DLoop (Smax⁡=3S_{\max}{=}3)742 / 4.76×1,321 / 4.36×2,573 / 4.34×4,803 / 4.22×8,294 / 3.89×13,020 / 3.46×
HumanEval
DFlash717 / 4.60×1,357 / 4.51×2,552 / 4.29×4,673 / 4.07×7,489 / 3.48×9,451 / 2.51×
+ DLoop779 / 4.98×1,454 / 4.83×2,648 / 4.45×4,791 / 4.17×7,668 / 3.56×9,104 / 2.42×
+ DLoop (Smax⁡=9S_{\max}{=}9)756 / 4.85×1,483 / 4.93×2,653 / 4.46×4,811 / 4.19×7,656 / 3.55×9,285 / 2.47×
+ DLoop (Smax⁡=3S_{\max}{=}3)737 / 4.72×1,446 / 4.80×2,636 / 4.43×4,841 / 4.21×7,711 / 3.58×9,545 / 2.54×
DSpark641 / 4.11×1,228 / 4.07×2,243 / 3.77×4,232 / 3.68×7,234 / 3.36×10,926 / 2.90×
+ DLoop714 / 4.58×1,326 / 4.41×2,451 / 4.12×4,317 / 3.76×7,297 / 3.39×10,825 / 2.88×
+ DLoop (Smax⁡=9S_{\max}{=}9)701 / 4.49×1,398 / 4.64×2,532 / 4.26×4,389 / 3.82×7,336 / 3.41×10,847 / 2.88×
+ DLoop (Smax⁡=3S_{\max}{=}3)682 / 4.37×1,280 / 4.25×2,459 / 4.13×4,329 / 3.77×7,471 / 3.47×11,105 / 2.95×

Lowering Smax⁡S_{\max} raises the throughput at c=32c = 32 for every draft model and benchmark, consistent with smaller differences in loop depth within a batch and less speculative work on a saturated target model. With Smax⁡=3S_{\max} = 3, DLoop exceeds the draft model at every concurrency for both draft models and both benchmarks. At low concurrency, a larger Smax⁡S_{\max} remains better, since the batch has spare capacity for additional drafting stages, which suggests choosing Smax⁡S_{\max} according to the load.

We have added this discussion and the new results to Section 4.2 of the main paper (Table 6).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

General Response to All Reviewers

We thank all reviewers for their thorough and constructive reviews. We appreciate the positive comments from the reviewers, which are summarized below.

  • The confidence gate adds almost no inference cost, and DLoop is a drop-in enhancement for publicly released draft models without architectural changes (Reviewer S7jE)
  • The idea is technically sound, the results demonstrate its effectiveness with meaningful improvement, and the paper is well written (Reviewer 1dkf)
  • The method is simple, lossless, and adds no parameters, it also benefits parallel draft models, and the evaluation is wide with a clear analysis and ablation (Reviewer Lgqu)

We have done our best to address every comment. Each weakness and question is answered with a new experiment or a clarification, and all of them are reflected in the revised manuscript. Each revised part is marked with the reviewer and item it answers. The clarifications and new experiments are summarized below. All section, table, and appendix numbers refer to the revised manuscript.

Clarifications and discussions

  • Added the discussion of how our gate function differs from LLMCad and DDD in Section 2 (W1 by Reviewer S7jE)
  • Added the discussion of how loop-aware training differs from the training-time test of EAGLE-3 in Section 3.3 (W1 by Reviewer S7jE)
  • Added the explanation of why previous adaptive draft length methods do not raise the speedup of the parallel draft model DSpark in Section 4.2 (W1 by Reviewer S7jE)
  • Clarified in Section 4.1 that the 1,024-token prefix is used only for the latency illustration in Figure 2 and Table A6 (W2-2 by Reviewer S7jE)
  • Moved the batch serving evaluation in SGLang from the appendix to Section 4.2 of the main paper, and added the discussion of choosing a cap on the loop depth according to the load (W2-3 by Reviewer S7jE, W1 by Reviewer Lgqu)
  • Stated the decoding setting in the abstract (W3 by Reviewer Lgqu)

New experiments

  • Added results on an A100 GPU for every draft model and target model in Appendix A.7 (W2-1 by Reviewer S7jE)
  • Added long-context results at 32K and 128K on LongSWE-Bench and GSM-∞ for every draft model in Appendix A.6 (W2-2 by Reviewer S7jE)
  • Added batch serving results in SGLang with a cap on the loop depth of each request, including the MTP module of Qwen3.5-9B with a small draft block, in Section 4.2 and Table 6 (W2-3 by Reviewer S7jE, W1 by Reviewer Lgqu)
  • Added an adaptive rule that sets the gate threshold gg automatically during decoding, and its comparison with fixed thresholds, in Appendix A.9.4 (Q1 by Reviewer 1dkf)
  • Added the runtime-based upper bound on the loop depth and the results for different bounds in Appendix A.9.5 (Q2 by Reviewer 1dkf)
  • Added the comparison with draft models fine-tuned for 2k2k and 3k3k blocks, with and without DLoop, in Appendix A.10.1 (W2 and Q1 by Reviewer Lgqu)
  • Added the comparison with larger draft depths for EAGLE-3 and MTP, with and without the loop, in Appendix A.10.1 (W2 and Q1 by Reviewer Lgqu)
  • Added the comparison between a fixed number of drafting stages and the gate in Appendix A.10.2 (Q2 by Reviewer Lgqu)
  • Added the full acceptance ratio at T=1T=1, with and without tree verification, in Appendix A.3 (W3 and Q3 by Reviewer Lgqu)

Detailed responses to each comment are provided in the individual responses.

AuthorsOpenReview ↗

Follow-up to All Reviewers

Dear Reviewers,

Thank you again for your time and thoughtful reviews.

As the discussion period ends in one day, we would like to kindly remind you of our responses. We would appreciate it if you could let us know whether our rebuttal has addressed your concerns. If so, we would be grateful if you would consider updating your assessment.

If any questions remain, we will be happy to clarify.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.