- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The core idea of LoopDraft is that the main role of a drafter is to approximate short-horizon distributions, so it does not need as many parameters. At the same time, increasing drafter depth may improve MAT or speedup. Therefore, the main question is whether a shallower drafter can get the same gains as a deeper one.
The authors design a recurrent drafter that can use fewer layers and still have a good acceptance rate or MAT. During this process, the authors observe diminishing returns and introduce a refinement budget to limit the number of inference steps through the shared block. They also add input-independent low-rank states to the recurrent structure and use a target-guided strategy to read and write these states. Meanwhile, LoopDraft also shares the target prefix during recursive inference. In the experiments, the best configuration, R5S4, gives roughly a 6.6%–7.4% improvement in decode-stage speedup at concurrency 1 and a 5.8%–6.2% improvement in MAT. It achieves a 6.4% improvement in speedup over DFlash on Qwen3-8B with an A30 at concurrency 1. The number of drafter parameters is reduced by a lot.
Critical strengths
- Further improvements to DFlash are among the directions of interest to the speculative decoding community.
- The authors show that, with a recurrent structure at concurrency 1, possibly using a Transformers implementation, LoopDraft has better MAT and speedup than DFlash.
- New drafter architectures are a direction that the community is interested in. LoopDraft reduces the number of drafter parameters by about three quarters. This may be useful in some scenarios.
- The paper is easy to follow and understand.
Critical weaknesses
- The main idea of the paper is that a drafter does not need as many parameters. The authors' recurrent structure is a little better than a non-recurrent structure such as DFlash. However, there is little improvement over DFlash v2 in Table 2: MAT increases from 7.32 to 7.37, which is less than 1%.
- In Section 4.4, the authors compare recurrent and non-recurrent structures and show curves of MAT and decode-stage speedup. However, the gains appear to be roughly 0.3 / 4.5 ≈ 6% (6.4% at T=0 and 7.4% at T=1). As a reader, I find these gains limited. The comparisons with Loop-5 throughout Section 4.5 show that the authors' designs are effective, but the overall gains over DFlash and DFlash v2 are still insufficient.
- The number of drafter parameters is reduced, but the currently measured gains from the full method with fewer parameters seem limited. Specifically, the experiments show roughly a 6.6%–7.4% improvement in decode-stage speedup.
Questions for authors
-
In Table 1, the proposed method, LoopDraft, achieves a higher speedup even when its MAT is slightly lower than DFlash's. For example, on MBPP at temperature 0, LoopDraft R3S4 has a lower MAT than DFlash-5L but a higher average speedup. This may suggest that LoopDraft's main gain comes from a reduction in T_draft (draft model inference time/latency). Although it has fewer drafter parameters, it uses recurrent inference, so T_draft does not necessarily decrease. Could the authors report T_draft to make this clearer to readers?
- Table 13 compares T_draft for Loop-5, R5S4, and R3S4, but why is DFlash-5L not included? Without it, we cannot clearly see where the gains come from.
-
The authors do not seem to provide a comparison of LoopDraft with DFlash/DFlash v2 at high concurrency. It would be useful to show how well the recurrent structure works at high concurrency and whether the gains are large, for example, in terms of decode-stage speedup.
2 comments
Thank you for the additional experiments and clarifications. The draft-latency breakdown and high-concurrency results address my questions about drafting cost and performance at higher concurrency. However, the source of the throughput gains over DFlash2 remains unclear to me, and the on-device motivation is not supported by the current experiments. Overall, I still find the contribution limited and will keep my rating.
Response to hFsi: parameter efficiency, latency attribution, and high-concurrency results
Thank you for recognizing technical soundness, clarity and parameter reduction. We address W1–W3 and Q1–Q3.
A. Contribution and deployment motivation (W1–W3)
The initial manuscript underemphasized our ultimate on-device goal and the additional drafter's memory budget. We focus on resource-constrained, low-concurrency interaction and have clarified this motivation in the revision. Current performance evidence comes from GPU experiments.
On DFlash2, MAT increases from 7.32 to 7.37 with 78.7% fewer draft parameters: 0.649B to 0.138B (Sec. 4.3, Table 2), preserving a strong baseline's proposal quality with a smaller drafter. At BF16 precision, these parameter reductions lower draft weight storage by approximately 1.02 GB for DFlash2 and 1.55 GB for Qwen3-8B (1.050B to 0.274B).
Our contribution concerns parameter, computation and proposal-quality tradeoffs. With matching target-feature taps and five layer evaluations, Loop-5 retains 97.7% of DFlash-5L's MAT and 99.9% of its speedup using 25.7% of its draft parameters (Appendix J, Table 12). Adding state control at the same five core calls increases MAT from 5.86 to 6.35. Against DFlash-5L, with five layer evaluations and matched target taps, R5S4 reduces draft parameters by 73.9% while improving decode-stage speedup by 6.4% at T=0 and 7.4% at T=1, calculated from unrounded means (Section 4.2, Table 1). These five-layer comparisons jointly demonstrate parameter efficiency and faster decoding.
The five-layer speed increment alone does not capture the gains at smaller parameter budgets. At T=0 (Table 1; App. E, Table 9), R5S4 uses a similar budget (0.274B vs 1-layer DFlash's 0.240B, +14.2%) for +99.1% acceptance/+61.3% speedup. Against 3-layer DFlash, R3S4 cuts parameters by 60.6% and improves acceptance/speedup by 26.5%/13.4% with matched layer evaluations and target taps (rounded means). The one-layer comparison uses more layer evaluations and taps; the three-/five-layer comparisons match both.
Loop-5 comparisons isolate the mechanism alongside direct DFlash/DFlash2 comparisons. App. D, Table 8 controls for extra training and parameters: continued training/matched one-state adapters give 4.25×/4.21× at T=0, versus R5S4's 4.55×.
B. Draft latency and the source of gains (Q1–Q2)
The original Appendix J, Table 13 reported 9.56, 6.65, and 10.12 ms draft latencies for Loop-5, R3S4, and R5S4 but omitted DFlash-5L. We have added its 10.27 ms latency to the revised table under the same configuration.
Latencies are five-task means on Qwen3-8B/A30, T=0, non-thinking, batch-one per worker, covering all steps of a proposal. Corresponding MAT and decode-stage speedups are in Table 1; speedups use target-only autoregressive decoding (AR) as the reference.
R3S4 reduces drafting cost: with three core calls, its draft latency is 35.2% lower than DFlash-5L's, while mean MAT remains comparable (6.02 versus 6.00). In the MBPP example from Table 1, MAT decreases from 5.48 to 5.42, yet speedup increases from 4.01× to 4.09×. This is consistent with the cycle-cost relation
\frac{T_{\mathrm{draft}}+T_{\mathrm{verify}}+T_{\mathrm{other}}}{\tau}.$$ A sufficiently lower cycle cost can outweigh a small MAT decrease. Table 13's 6.65 ms is R3S4's five-task mean draft latency; Table 1 reports the MBPP-specific acceptance and speedup. **R5S4 improves proposal quality at similar drafting cost:** 10.12 ms versus DFlash-5L's 10.27 ms, with MAT increasing from 6.00 to 6.35 and speedup from 4.27× to 4.55×. Moreover, adding state control to Loop-5 increases draft latency from 9.56 to 10.12 ms while improving speed. Additional accepted tokens compensate for state-control overhead, improving speed at comparable drafting cost. Relative to Loop-5, R5S4 takes 5.9% longer to draft but advances 8.4% more tokens per verification. Its speed gain therefore cannot be explained by a reduction in draft latency alone. **C. Higher-concurrency comparisons (Q3)** Measured five-task mean serving throughput speedups at T=0, non-thinking, match target, protocol and client concurrency within each comparison, including its AR reference. | Client concurrency | DFlash | R5S4 | |---|---:|---:| | 1 | 4.50× | 4.80× | | 4 | 3.77× | 4.02× | | 8 | 3.76× | 4.00× | | 16 | 3.27× | 3.49× | | 32 | 2.36× | 2.51× | Target: Qwen3-8B; H100/SGLang. | Client concurrency | DFlash2 | DFlash2 + R5S4 | |---|---:|---:| | 1 | 4.20× | 4.50× | | 16 | 2.80× | 2.97× | | 32 | 1.95× | 2.05× | Target: Qwen3.5-4B; H100/SGLang. Relative gains over DFlash2 are approximately 7.1%, 6.1%, and 5.1%, computed from the displayed values, alongside 78.7% fewer draft parameters. Table 2 already reports DFlash2's MAT and parameter efficiency at client concurrency 16; these additional measurements report serving throughput. Although speedups over AR decrease with concurrency, R5S4 retains gains over both original architectures at every tested level.