acceptodds
Under review as a conference paper at ICLR 2027

LoopDraft: Target-Guided Multi-State Recurrence for Speculative Decoding

Abstract

Increasing a speculative drafter’s physical depth can improve proposal quality, but also ties computation to independently parameterized layers. A drafter need not match the target model’s full capability: it only needs to approximate short-horizon predictive distributions. This raises the question of whether the benefit of depth requires distinct layers or can instead be recovered through iterative refinement. We introduce LoopDraft, which reuses a shared computation block across an evolving latent state, recovering much of a deeper drafter’s benefit with about one quarter of its draft parameters. However, further recurrence quickly encounters diminishing returns, motivating better use of a fixed refinement budget. LoopDraft therefore maintains multiple persistent states and uses verified-prefix target features to guide state access and retention while preserving shared-core recurrent computation. On Qwen3-8B, R5S4 reaches 4.55× speedup with a mean acceptance length of 6.35, improving over a local five-layer DFlash baseline by 6.4% while reducing draft parameters by 73.9%. On Qwen3.5-4B, the same design improves acceptance length across DFlash, DSpark, and DFlash2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Large Language Model Inference, Efficient Decoding, Recurrent Refinement, Draft Models, Test-Time Computation, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hFsiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The core idea of LoopDraft is that the main role of a drafter is to approximate short-horizon distributions, so it does not need as many parameters. At the same time, increasing drafter depth may improve MAT or speedup. Therefore, the main question is whether a shallower drafter can get the same gains as a deeper one.

The authors design a recurrent drafter that can use fewer layers and still have a good acceptance rate or MAT. During this process, the authors observe diminishing returns and introduce a refinement budget to limit the number of inference steps through the shared block. They also add input-independent low-rank states to the recurrent structure and use a target-guided strategy to read and write these states. Meanwhile, LoopDraft also shares the target prefix during recursive inference. In the experiments, the best configuration, R5S4, gives roughly a 6.6%–7.4% improvement in decode-stage speedup at concurrency 1 and a 5.8%–6.2% improvement in MAT. It achieves a 6.4% improvement in speedup over DFlash on Qwen3-8B with an A30 at concurrency 1. The number of drafter parameters is reduced by a lot.

Critical strengths
  1. Further improvements to DFlash are among the directions of interest to the speculative decoding community.
  2. The authors show that, with a recurrent structure at concurrency 1, possibly using a Transformers implementation, LoopDraft has better MAT and speedup than DFlash.
  3. New drafter architectures are a direction that the community is interested in. LoopDraft reduces the number of drafter parameters by about three quarters. This may be useful in some scenarios.
  4. The paper is easy to follow and understand.
Critical weaknesses
  1. The main idea of the paper is that a drafter does not need as many parameters. The authors' recurrent structure is a little better than a non-recurrent structure such as DFlash. However, there is little improvement over DFlash v2 in Table 2: MAT increases from 7.32 to 7.37, which is less than 1%.
  2. In Section 4.4, the authors compare recurrent and non-recurrent structures and show curves of MAT and decode-stage speedup. However, the gains appear to be roughly 0.3 / 4.5 ≈ 6% (6.4% at T=0 and 7.4% at T=1). As a reader, I find these gains limited. The comparisons with Loop-5 throughout Section 4.5 show that the authors' designs are effective, but the overall gains over DFlash and DFlash v2 are still insufficient.
  3. The number of drafter parameters is reduced, but the currently measured gains from the full method with fewer parameters seem limited. Specifically, the experiments show roughly a 6.6%–7.4% improvement in decode-stage speedup.
Questions for authors
  1. In Table 1, the proposed method, LoopDraft, achieves a higher speedup even when its MAT is slightly lower than DFlash's. For example, on MBPP at temperature 0, LoopDraft R3S4 has a lower MAT than DFlash-5L but a higher average speedup. This may suggest that LoopDraft's main gain comes from a reduction in T_draft (draft model inference time/latency). Although it has fewer drafter parameters, it uses recurrent inference, so T_draft does not necessarily decrease. Could the authors report T_draft to make this clearer to readers?

    • Table 13 compares T_draft for Loop-5, R5S4, and R3S4, but why is DFlash-5L not included? Without it, we cannot clearly see where the gains come from.
  2. The authors do not seem to provide a comparison of LoopDraft with DFlash/DFlash v2 at high concurrency. It would be useful to show how well the recurrent structure works at high concurrency and whether the gains are large, for example, in terms of decode-stage speedup.

2 comments
AuthorsOpenReview ↗

Response to hFsi: parameter efficiency, latency attribution, and high-concurrency results

Thank you for recognizing technical soundness, clarity and parameter reduction. We address W1–W3 and Q1–Q3.

A. Contribution and deployment motivation (W1–W3)

The initial manuscript underemphasized our ultimate on-device goal and the additional drafter's memory budget. We focus on resource-constrained, low-concurrency interaction and have clarified this motivation in the revision. Current performance evidence comes from GPU experiments.

On DFlash2, MAT increases from 7.32 to 7.37 with 78.7% fewer draft parameters: 0.649B to 0.138B (Sec. 4.3, Table 2), preserving a strong baseline's proposal quality with a smaller drafter. At BF16 precision, these parameter reductions lower draft weight storage by approximately 1.02 GB for DFlash2 and 1.55 GB for Qwen3-8B (1.050B to 0.274B).

Our contribution concerns parameter, computation and proposal-quality tradeoffs. With matching target-feature taps and five layer evaluations, Loop-5 retains 97.7% of DFlash-5L's MAT and 99.9% of its speedup using 25.7% of its draft parameters (Appendix J, Table 12). Adding state control at the same five core calls increases MAT from 5.86 to 6.35. Against DFlash-5L, with five layer evaluations and matched target taps, R5S4 reduces draft parameters by 73.9% while improving decode-stage speedup by 6.4% at T=0 and 7.4% at T=1, calculated from unrounded means (Section 4.2, Table 1). These five-layer comparisons jointly demonstrate parameter efficiency and faster decoding.

The five-layer speed increment alone does not capture the gains at smaller parameter budgets. At T=0 (Table 1; App. E, Table 9), R5S4 uses a similar budget (0.274B vs 1-layer DFlash's 0.240B, +14.2%) for +99.1% acceptance/+61.3% speedup. Against 3-layer DFlash, R3S4 cuts parameters by 60.6% and improves acceptance/speedup by 26.5%/13.4% with matched layer evaluations and target taps (rounded means). The one-layer comparison uses more layer evaluations and taps; the three-/five-layer comparisons match both.

Loop-5 comparisons isolate the mechanism alongside direct DFlash/DFlash2 comparisons. App. D, Table 8 controls for extra training and parameters: continued training/matched one-state adapters give 4.25×/4.21× at T=0, versus R5S4's 4.55×.

B. Draft latency and the source of gains (Q1–Q2)

The original Appendix J, Table 13 reported 9.56, 6.65, and 10.12 ms draft latencies for Loop-5, R3S4, and R5S4 but omitted DFlash-5L. We have added its 10.27 ms latency to the revised table under the same configuration.

Latencies are five-task means on Qwen3-8B/A30, T=0, non-thinking, batch-one per worker, covering all steps of a proposal. Corresponding MAT and decode-stage speedups are in Table 1; speedups use target-only autoregressive decoding (AR) as the reference.

R3S4 reduces drafting cost: with three core calls, its draft latency is 35.2% lower than DFlash-5L's, while mean MAT remains comparable (6.02 versus 6.00). In the MBPP example from Table 1, MAT decreases from 5.48 to 5.42, yet speedup increases from 4.01× to 4.09×. This is consistent with the cycle-cost relation

\frac{T_{\mathrm{draft}}+T_{\mathrm{verify}}+T_{\mathrm{other}}}{\tau}.$$ A sufficiently lower cycle cost can outweigh a small MAT decrease. Table 13's 6.65 ms is R3S4's five-task mean draft latency; Table 1 reports the MBPP-specific acceptance and speedup. **R5S4 improves proposal quality at similar drafting cost:** 10.12 ms versus DFlash-5L's 10.27 ms, with MAT increasing from 6.00 to 6.35 and speedup from 4.27× to 4.55×. Moreover, adding state control to Loop-5 increases draft latency from 9.56 to 10.12 ms while improving speed. Additional accepted tokens compensate for state-control overhead, improving speed at comparable drafting cost. Relative to Loop-5, R5S4 takes 5.9% longer to draft but advances 8.4% more tokens per verification. Its speed gain therefore cannot be explained by a reduction in draft latency alone. **C. Higher-concurrency comparisons (Q3)** Measured five-task mean serving throughput speedups at T=0, non-thinking, match target, protocol and client concurrency within each comparison, including its AR reference. | Client concurrency | DFlash | R5S4 | |---|---:|---:| | 1 | 4.50× | 4.80× | | 4 | 3.77× | 4.02× | | 8 | 3.76× | 4.00× | | 16 | 3.27× | 3.49× | | 32 | 2.36× | 2.51× | Target: Qwen3-8B; H100/SGLang. | Client concurrency | DFlash2 | DFlash2 + R5S4 | |---|---:|---:| | 1 | 4.20× | 4.50× | | 16 | 2.80× | 2.97× | | 32 | 1.95× | 2.05× | Target: Qwen3.5-4B; H100/SGLang. Relative gains over DFlash2 are approximately 7.1%, 6.1%, and 5.1%, computed from the displayed values, alongside 78.7% fewer draft parameters. Table 2 already reports DFlash2's MAT and parameter efficiency at client concurrency 16; these additional measurements report serving throughput. Although speedups over AR decrease with concurrency, R5S4 retains gains over both original architectures at every tested level.

Thank you for the additional experiments and clarifications. The draft-latency breakdown and high-concurrency results address my questions about drafting cost and performance at higher concurrency. However, the source of the throughput gains over DFlash2 remains unclear to me, and the on-device motivation is not supported by the current experiments. Overall, I still find the contribution limited and will keep my rating.

Reviewer_BmR8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoopDraft hypothesizes that the draft model size can be traded off with looping and refining the states of the draft model to align closely with the verifier. The main proposal is to trade off depth of draft model which is slower to run with recursive refinement using a smaller draft.

In order to align the drafter close to the expected states of the verifier the authors propose the following:

  1. Use KV projections of the verifier context as KV for the drafter.
  2. Introduce several learnable memory states for the context to select the correct state during refinement.
  3. Introduce read and write adapters to control the flow of memory states.
  4. Target guided persistence gate and state access weights

The results support two distinct claims:

  • Controlled memory improves refinement
  • Better state use can replace computation

The 4.55x is relative to ordinary target-only decoding. The improvement over the existing DFlash is marginal.

Critical strengths
  1. Lower parameter configuration (R5S4) nearly matches five-layer DFlash while reducing draft parameters from 1.05B to 0.27B.
  2. A useful lower-compute configuration (R3S4) demonstrated approximately the five-layer baseline’s acceptance length using three core evaluations.
  3. Ablations on various additional components iterating from static to recurrent (R5) and also showing S=1 to S=4 switch demonstrates impact of each contribution.
Critical weaknesses
  1. The main speed results use Qwen3-8B, A30 GPUs, and batch-one decoding per worker. This might be too small of a use case for real benefit analysis.
  2. The gain over DFlash is marginal. The parameter reduction is the larger contribution.
  3. Ablations show that adaptive access and retention help, but do not establish what the states store or why target guidance helps particular tokens.
Questions for authors
  1. Can you elaborate on what the states encode?
  2. Share analysis of the gates. As of now it seems similar to gates in RNNs or LSTMs.
  3. Why use gates at all and why not preserve old states and attend over them instead?
1 comment
AuthorsOpenReview ↗

Response to BmR8: deployment scope, state representations, and gate analysis

Thank you for recognizing parameter efficiency, lower computation and ablations. The manuscript underemphasized our resource-constrained, low-concurrency motivation, ultimately on-device deployment. We have clarified this in the revision; measurements use GPUs.

A. Scope and contribution (W1-2)

Table 1 (Qwen3-8B/A30, five-task means) shows that, with five layer evaluations and matched target taps, R5S4 reduces draft parameters by 73.9% while improving acceptance and speed over DFlash-5L (+6.4%/+7.4% speed at T=0/1, unrounded means). R3S4 matches five-layer acceptance with three core calls (τ=6.02\tau=6.02 vs 6.00). Speedups are vs target-only autoregressive decoding (AR); τ\tau includes the anchor. Latency depends on computation and acceptance.

At T=0 (Table 1; App. E, Table 9), R5S4 uses a similar budget (0.274B vs 1-layer DFlash's 0.240B, +14.2%) for +99.1% acceptance/+61.3% speedup. Against 3-layer DFlash, R3S4 cuts parameters by 60.6% and improves acceptance/speedup by 26.5%/13.4% with matched layer evaluations and target taps (rounded means).

Broader evaluation:

  • Sec. 4.3/Table 2: Qwen3.5-4B, DFlash/DSpark/DFlash2, both thinking regimes, vLLM client concurrency 16. R5S4 improves all six mean-acceptance comparisons with 71.1-79.0% fewer parameters.
  • Added Qwen3-8B/H100/SGLang, T=0: at concurrency 16/32, R5S4 gives 3.49x/2.51x vs DFlash's 3.27x/2.36x. Concurrency denotes clients.
  • H100/Transformers, T=0, thinking off: five-task means are 4.50x/4.50x/4.80x for DFlash/Loop-5/R5S4, matching the ordering on A30 (Table 1).
  • Qwen3.6-35B-A3B MoE (35B total/~3B active): H100/vLLM, T=0, non-thinking; mean τ\tau is 6.483 vs Loop-5's 5.603 and local DFlash-6L's 6.322, supporting proposal-quality transfer.

B. What states encode (W3; Q1)

States are input-dependent proposal representations, initialized as Ys,k(0)=Xk(0)Y^{(0)}_{s,k}=X^{(0)}_k. They persist within one proposal, resetting across decoding rounds. Their update is

Ys,k(r+1)=gr,s,kYs,k(r)+ΔYs,k(r)Y^{(r+1)}_{s,k}=g_{r,s,k}Y^{(r)}_{s,k}+\Delta Y^{(r)}_{s,k}.

Each state carries retained input components and refinement updates via stream-specific writes and retention.

Checkpoint analysis shows divergence after write-back, including in the low-rank states read by the core. Read mixtures vary by position, step and input. Dominant streams change; some positions use balanced mixtures. States carry distributed, input-dependent refinement histories used in later computation.

C. Gate behavior and target guidance (W3; Q2)

Gating has precedent in RNNs/LSTMs. We control access before low-rank fusion and old-state retention after writing along refinement depth, each with one scalar per position/stream.

Normalize access as p=a/Sp=a/S, since ∑sas=S\sum_s a_s=S. Identical initial streams have a=1a=1 (p=0.25p=0.25 for S=4). After writing, access varies with input, position and step. Retention differs across streams at fixed position/step. We observe retention followed by attenuation within streams, with patterns depending on the input rather than a fixed schedule.

Joint traces show high-read/high-retain, low-read/low-retain and high-read/low-retain cases. Some low-read/high-retain states are read more strongly later; some strongly read states have low retention, separating current access from later preservation. We inspect both ∥gY∥\|gY\| and ∥ΔY∥\|\Delta Y\| to measure retained-state and new-update contributions. Across analyzed samples, changes in access/retention accompany proposal corrections and better acceptance at corresponding positions.

Controllers use the latest verified-prefix target representation, current state, position and step; retention also uses the write representation. No extra target pass is needed. Removing controller target input changes read/retention behavior at some positions. These traces explain behavior; five-call ablations test performance:

Variantτ\tauSpeedup
Loop-55.864.27x
Static S=45.924.22x
Retention only6.054.28x
Access only6.174.34x
Local dynamic, no target control6.174.35x
Full target-guided6.354.55x

The core remains target-conditioned in the local dynamic variant. Parameter-matched one-state and continued-training controls give 5.90/4.21x and 5.90/4.25x (Appendix D).

D. Why not attend over old states? (Q3)

Access selects content; write-back and retention govern state updates and preservation. Later use of retained states (C) illustrates this distinction. The gate scales old state, while the new update enters additively.

Keeping all past states increases storage and reading costs as history grows; fixed-capacity attention still needs an update policy. Our S-state interface uses 2NKS control scalars per step without multiplying R core calls by S. Modules add 3.70M parameters; S=4 states occupy 0.50 MiB/sequence at K=16, H=4096, BF16 (Appendix H). Attention is a reasonable alternative; our design explicitly controls reads and retention at a fixed state budget.

Reviewer_TsFbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a speculative decoding method, operating in the same setting as DFlash, that performs diffusion-like iterative refinement using shared weights to generate draft tokens. Multiple stream states are maintained per token position, and a gating network determines which states to read from and which to write to at each refinement step. These components together yield an increase in mean accepted tokens even with a reduced parameter count. The method demonstrates competitive performance on small models (Qwen3.5-4B) compared to DFlash-5L and EAGLE-3.

Critical strengths

The combination of loop-based iterative refinement (analogous to diffusion denoising) and target-guided state control is an interesting design. While each component individually has precedent, their joint application to speculative decoding drafters is novel.

Parameter efficiency is better than DFlash, although it is not clear if this is critical when model size is already very large in many cases.

Critical weaknesses

The design is largely incremental. It is a combination of existing ideas.

The paper doesn't explain why it works well.

The paper is written in a way that is fairly hard to understand.

The evaluation is incomplete in several respects. There are many other baselines that claim to be much faster than DFlash including XPress, DSpark, DFlash2, etc. The paper fails to compare against them.

Only temp =0 and 1 are evaluated.

It only showed Qwen3.5-4B and Qwen3-8B. For Qwen3-8B, only hyperparameter search was done but comparison against baselines was missing.

The model size evaluated is very small, only 4B. It might be that when model size is large, the relative performance gain diminishes because the extra cost of DFlash becomes relatively small and almost negligible.

It was hard to find any insight in the hyperparameter setting.

Questions for authors
  1. How does it compare against other baselines that are more recent than DFlash?

  2. How does it perform with larger batch sizes? Speculative decoding is known to be less effective with larger batches. Please draw the latency-throughput curve while varying the batch size. Does this have benefit with larger batch sizes against other baselines?

  3. Does it extend to larger and more recent models?

  4. How does it perform on different GPUs?

  5. Why is reusing weights critical? The weights are much smaller than the model itself. If the model size is large, does it still show benefit? It might be that when model size is large, the relative performance gain diminishes because the extra cost of DFlash becomes relatively small and almost negligible. If so, the evaluation is only showing cherry-picked results.

6 . How does this perform with different temperature settings?

1 comment
AuthorsOpenReview ↗

Response to TsFb: parameter efficiency, recent baselines, and extended evaluation

Thank you for the review.

A. Motivation, mechanism and insight (W1-3, W6, W8; Q5)

We study resource-constrained draft budgets for low-concurrency interaction, ultimately on-device deployment.

The incremental-design criticism overlooks the separation of parameters, computation and state capacity (Fig. 2, Secs. 3-4):

  • With matched target taps and five layer evaluations, Loop-5 retains 97.7% of DFlash-5L's acceptance and 99.9% of its speedup using 25.7% of its parameters.
  • More recurrence saturates: 5 -> 15 steps raises τ\tau from 5.86 to 6.06 but lowers speedup from 4.27x to 3.58x.
  • mHC-inspired streams use low-rank read/write and target-guided access/retention at fixed core-call count.

Each step reads states, runs one shared core and updates states. Checkpoint traces show divergent streams, input-dependent reads and later use of retained states.

At five calls, static S=4 gives τ=5.92\tau=5.92; retention-only 6.05; access-only 6.17; full control 6.35/4.55x. Removing controller target input, while retaining target-conditioned core computation, gives 6.17/4.35x. Parameter-matched one-state and continued-training controls stay near Loop-5 (Appendix D). Verified-prefix control needs no extra target pass. States persist within one proposal.

The claim that Qwen3-8B lacks baseline comparisons is incorrect: Table 1/Sec. 4.2 compare DFlash and EAGLE-3 at T=0/1. With five layer evaluations and matched taps on A30, R5S4 reduces parameters from 1.050B to 0.274B (-73.9%), improving mean speedup from 4.27x to 4.55x at T=0 and 3.64x to 3.91x at T=1. Speedups are vs target-only autoregressive decoding (AR); τ\tau includes the anchor; counts exclude shared embeddings/LM head.

S=8 raises τ\tau to 6.44 and draft latency from 10.12 to 10.75 ms, lowering speedup to 4.48x. R3S4 reaches 4.41x/τ=6.02\tau=6.02 vs DFlash's 6.00 with three calls, at 6.65 ms vs Loop-5's 9.56 ms. State use can replace some computation. Depth/width sweeps also report slower configurations. Decoding speed reflects both proposal cost and accepted tokens, including for larger targets (C).

At T=0 (Table 1; App. E, Table 9), R5S4 uses a similar budget (0.274B vs 1-layer DFlash's 0.240B, +14.2%) for +99.1% acceptance/+61.3% speedup. Against 3-layer DFlash, R3S4 cuts parameters by 60.6% and improves acceptance/speedup by 26.5%/13.4% with matched layer evaluations and target taps (rounded means).

B. Recent baselines (W4; Q1)

The claim that DSpark/DFlash2 comparisons are missing is incorrect: The original submission includes them (Sec. 4.3, Table 2, p. 8), with native mechanisms. On Qwen3.5-4B, R5S4 improves mean τ\tau for DFlash/DSpark/DFlash2 in all six regime comparisons with 71.1-79.0% fewer parameters. DSpark: 5.57/7.06 (thinking/non-thinking).

XPress reports no Qwen3.5-4B result, so we reproduced it under the same five-task T=0 non-thinking protocol: τ=7.34\tau=7.34. Our DFlash2+R5S4 gives 7.37 vs the local six-layer DFlash2's 7.32, with 0.138B vs 0.649B draft parameters (-78.7%). The 78.7% reduction uses local DFlash2 as the parameter reference.

C. Larger/newer targets (W7; Q3)

Beyond the models/backbones in B, we tested Qwen3.6-35B-A3B (35B total/~3B active). H100/vLLM, T=0, non-thinking:

DrafterGSM8KHumanEvalBBHMATH-500MBPPMT-BenchMean
Local DFlash-6L8.449.075.367.683.873.516.322
Loop-57.908.143.727.043.563.265.603
Multi-stream8.609.205.507.904.003.706.483

Multi-stream gains 15.70% over Loop-5 and 2.56% over DFlash-6L, exceeding the latter on every task: proposal-quality transfer to an MoE with more total parameters.

D. Concurrency (Q2)

Table 2 already uses vLLM client concurrency 16. Qwen3-8B/H100/SGLang, T=0; five-task means: GSM8K, MATH-500, HumanEval, MBPP, MT-Bench. Indices: normalize each task to its concurrency-1 AR (=100), then average. Speedups use matched-concurrency AR.

Conc.AR indexDFlash indexR5S4 indexDFlash speedR5S4 speed
1100.0450.0480.04.50x4.80x
4350.01319.51407.03.77x4.02x
8650.02444.02600.03.76x4.00x
161150.03760.54013.53.27x3.49x
321900.04484.04769.02.36x2.51x

Throughput rises; AR-relative gains shrink. R5S4 beats DFlash at all tested client concurrencies.

E. GPUs (Q4)

Qwen3-8B, T=0, non-thinking; five-task mean speedups vs AR (Transformers):

GPUDFlash-5LLoop-5R5S4
A304.27x4.27x4.55x
H1004.50x4.50x4.80x

Acceptance is generally stable across GPUs at fixed checkpoints/samples/protocols. Loop-5 matches DFlash; R5S4 gains on both GPUs.

F. Temperatures (W5; Q6)

DFlash/XPress report T=0/1 (Table 1), DFlash2 evaluates T=0/1 (Table 2); DSpark uses T=1 offline (Sec. 4.1). Our T=0 (greedy)/T=1 (sampling) protocol follows these evaluations. With thinking off, R3S4/R5S4 improve acceptance and speed over DFlash-5L at both temperatures (A). R5S4 gives 4.55x/3.91x at T=0/1 (+6.4%/7.4%, unrounded means).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript revisions: clarified motivation, reorganized existing evidence, and supplementary validation

We have uploaded a revised manuscript addressing the reviews. The changes below distinguish clarification and reorganization of original evidence from additional validation. Section and table references refer to the revised manuscript unless explicitly marked as original.

  • Clarified motivation and comparison emphasis (Abstract, Introduction and Section 4.1). We make explicit the draft-parameter budget relevant to resource-constrained, on-device interaction, while retaining GPU-based evaluation. The presentation now emphasizes both performance near a small parameter budget and comparisons at matched layer-evaluation counts and target-feature interfaces.

  • Brought existing shallow-baseline evidence into the main presentation (Figure 1, Table 1 and Sections 4.2/4.4). DFlash-1L and DFlash-3L T=0 five-task means were already in the original Appendix E, Table 9; the original Appendix J, Table 12 also compared physical and recurrent depth. Table 1 now includes these baselines and expands task-level reporting at T=0/1. Figure 1 and the results discussion highlight the one-, three- and five-layer comparisons. The one-layer comparison uses a similar parameter budget with more layer evaluations and target taps; the three-/five-layer comparisons match layer-evaluation counts and target taps. These changes expose the original method's parameter and computation tradeoffs more clearly.

  • Retained existing transfer comparisons (Section 4.3, Table 2). Comparisons across DFlash, DSpark and DFlash2 on Qwen3.5-4B were already in the original submission and are retained. Section 4.1 now explicitly directs readers to this evaluation. These are not newly introduced baseline experiments.

  • Completed draft-latency reporting (Appendix J, Table 13; Appendix L.1). We added the DFlash-5L draft-latency measurement omitted from the original Table 13. The accompanying analysis distinguishes the lower drafting cost of R3S4 from the improved acceptance of R5S4 at comparable drafting cost.

  • Added supplementary validation of the same method (Appendix L.2-L.5). These sections report the Qwen3.6-35B-A3B MoE target, higher client concurrency, cross-GPU decoding and a local XPress reproduction. Serving-throughput measurements and Transformers decode-stage measurements use their respective matched AR references and are defined separately. XPress is also added to Related Work and the bibliography.

  • Added qualitative checkpoint analysis (Appendix M). We describe stream divergence, input-dependent access and retention, and later use of preserved states, supplementing the original controlled ablations with observations of internal behavior.

The core architecture, training objective and technical progression remain unchanged: shared refinement recovers much of physical depth's benefit; further recurrence has diminishing returns; persistent states and target-guided access/retention improve the use of a fixed refinement budget. Detailed responses are provided separately for each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.