- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Wavefront Decoding (WFD), a training-free self-speculative decoding method for looped language models. WFD uses shallow recurrence outputs as drafts and batches token states at different positions and recurrence depths into a diagonal wavefront. This overlaps drafting, recurrent computation, and full-depth verification within the same recurrent-block calls. Rejected drafts are corrected using full-depth predictions. The paper provides a latency model showing that WFD reaches the asymptotic latency of Draft-then-Verify (DtV) with a finite wavefront. Experiments on Ouro-2.6B and Huginn-3.5B report 2.42x and 3.54x speedups over autoregressive decoding and a 1.27x improvement over DtV.
Critical strengths
WFD is a natural and well motivated schedule for looped models. It uses shallow recurrence outputs as drafts while exploiting weight sharing to batch states at different recurrence depths. This removes the phase boundary in DtV without requiring another model or additional training. The construction applies to both full-stack and P/R/C looped architectures.
Critical weaknesses
- The evidence for realistic serving workloads or at larger model scales is limited. (1) The main experiments use two 2.6B/3.5B models on a single A6000, mostly at user batch size 1. The appendix shows that speedup decreases as batch size grows, particularly without cross-recurrence KV sharing which is model-dependent, approximate and could not be applied to Ouro without harming accuracy. (2) Fixed-batch experiments also omit several effects that interact directly with the proposed schedule, such as continuous batching, heterogeneous output lengths, and rejection-induced wavefront flushing. These effects can reduce the effective recurrent batch size or introduce bubbles and queueing between the P/R/C stages. (3) No larger or distributed target model is evaluated, so it is unclear whether mixed-depth recurrent calls remain nearly as cheap as single-position calls once execution becomes compute- or communication-bound. Since WFD is primarily a serving schedule, these missing evidence substantially limit the practical conclusions.
- The evaluation lacks conventional speculative decoding with an external draft model as baselines. The paper presents the lack of auxiliary training and draft-model weights as an advantage, but only compares WFD with AR and the self-speculative DtV baseline. It therefore remains unclear whether WFD offers a favorable throughput-memory trade-off against a small external drafter, or whether its training-free design sacrifices attainable speed. A comparison under similar memory and quality constraints is needed to assess the practical value of this design choice.
Questions for authors
- Was an external draft model considered or tested with either target model? If so, how did its throughput, memory use, and acceptance rate compare with WFD? If not, is there a technical reason this baseline is unsuitable for looped targets?
- In the batch-size experiments, how much of the speedup reduction comes from better weight amortization in the AR baseline, and how much comes from WFD-specific KV traffic or increased recurrent-block latency?
- Can the authors provide a continuous-batching experiment using variable prompt and output lengths, preferably reporting throughput and p95/p99 latency?
- Can the authors provide results on a larger or distributed looped model, or profiling evidence showing that mixed-depth recurrent calls remain bandwidth-bound as model size and effective batch size increase?
9 comments
Question 3
To the best of our knowledge, none of the existing serving engines supports Huginn. We therefore built a custom serving engine that supports full CUDA-graph execution with continuous batching, natural rejection and scheduling.
To assess how WFD performs under more realistic serving conditions (variable prompt and output lengths), we evaluated it on LongBench [5] with the custom engine at a concurrency of 8.
Table C. Serving performance of Ouro-2.6B-Thinking on 280 LongBench samples at a concurrency of 8. Prompt lengths range from 2048 to 16384 tokens (mean 7431, std 5365).
| Metric | AR | DtV | WFD |
|---|---|---|---|
| tok/s ( AR) | 20.72 | 25.15 (1.21$\times$) | 26.99 (1.30$\times$) |
| Decode tok/s ( AR) | 22.69 | 28.04 (1.24$\times$) | 30.44 (1.34$\times$) |
| α | – | 0.77 | 0.77 |
| TPOT p95/p99 (ms) | 114.2/120.0 | 108.5/122.8 | 98.7/106.6 |
| E2E latency p95/p99 (s) | 207.6/230.0 | 174.2/210.8 | 154.0/172.3 |
| Effective concurrency | 2.0 | 2.0 | 2.2 |
Table D. Serving performance of Huginn-3.5B on 100 LongBench samples at a concurrency of 8. Prompt lengths range from 193 to 2048 tokens (mean 1598, std 420).
| Metric | AR | DtV | WFD w/ KV sharing |
|---|---|---|---|
| tok/s ( AR) | 19.12 | 30.11 (1.57$\times$) | 42.17 (2.21$\times$) |
| Decode tok/s ( AR) | 28.34 | 60.48 (2.13$\times$) | 139.91 (4.94$\times$) |
| α | – | 0.91 | 0.97 |
| TPOT p95/p99 (ms) | 339/352 | 326/365 | 245/399 |
| E2E latency p95/p99 (s) | 59.5/63.7 | 40.5/46.8 | 32.4/40.9 |
| Effective concurrency | 6.0 | 6.0 | 7.6 |
Table C shows that WFD outperforms DtV on 2K–16K-token contexts at a concurrency of 8. Huginn was trained with (and defaults to) a 4096-token context length, so we evaluated it on contexts of up to 2K tokens in Table D. On Ouro, WFD shows a lower speedup than in our main results, but still outperforms DtV. The reduced gain mainly stems from the lower acceptance rate on LongBench and longer contexts. On Huginn, WFD achieves 4.94$\times$ decode speedup with KV sharing, demonstrating its effectiveness in a serving setting. WFD also shows lower E2E p95/p99 latency on both models.
Question 4
We acknowledge your concerns about scalability to larger models and batch sizes. To the best of our knowledge, no larger publicly available checkpoints exist for either model type (full-stack and P/R/C). Ouro-2.6B and Huginn-3.5B have hidden dimensions of 2048 and 5280, and loop-unrolled parameter counts of ~10B and ~54B, respectively. These models are therefore comparable in per-token compute and weight traffic to dense models with tens of billions of parameters, such as Llama-2-13B or Qwen3-32B.
To assess wavefront decoding’s scalability to larger hidden dimensions, we instantiated dummy-weight models with dimensions adapted from Kimi K2 (hidden dimension of 7168, 56 query heads, 4:1 GQA, head dimension of 128, FFN dimension of 18432, vocabulary size of 163840), following the Ouro (full-stack) and Huginn (P/R/C) architectures.
Table E. Speedup of each decoding method on dummy-weight models (batch 1, 1024-token
prefill, 512 decode steps, fixed at the overall rates from the Spec-Bench results).
| AR | DtV | WFD | |
|---|---|---|---|
| Dummy-Huginn-Big | 1.00$\times$ | 2.89$\times$ | 4.28$\times$ |
| Dummy-Ouro-Big | 1.00$\times$ | 1.79$\times$ | 3.04$\times$ |
Table E shows the speedup of WFD on dummy-weight models with fixed acceptance rates. WFD achieves higher speedups than in our main results because the larger hidden dimension and GQA ratio shift the memory-traffic bottleneck from attention to the FFN, which favors WFD even without cross-recurrence KV sharing.
We also provide an arithmetic-intensity analysis in Appendix C.1. In short, with or without cross-recurrence KV sharing, the decoding phase remains memory-bandwidth-bound for practical batch sizes (up to ~200).
Response to Weakness 2, Question 1
We sincerely appreciate the insightful question. As you pointed out, requiring neither additional training nor a separate drafter is an advantage of WFD. Nevertheless, we agree that comparing WFD against conventional speculative decoding (SD) is important for assessing its practical serving performance.
To the best of our knowledge, no external draft model has been specifically designed or trained for Ouro or Huginn. To provide an initial comparison, we conducted additional experiments using SmolLM2-135M [6], a publicly available model that shares Ouro's tokenizer, as the drafter (Table F). We selected SmolLM2-135M because it outperformed SmolLM2-360M in our experiments. For Huginn, which uses its own separately trained tokenizer, no suitable public drafter is available.
Table F. Throughput on the custom serving engine across Spec-Bench tasks (concurrency = 1). α denotes the acceptance rate. All other settings follow the main results.
| Mode | Huginn-3.5B | Ouro-2.6B | ||||
|---|---|---|---|---|---|---|
| Throughput (tok/s) | Speedup | Alpha | Throughput (tok/s) | Speedup | Alpha | |
| vLLM AR | - | - | - | 27.46 | - | - |
| AR | 5.67 | 1.00 | 28.01 | 1.00 | ||
| DtV | 15.35 | 2.71 | 0.93 | 52.09 | 1.86 | 0.91 |
| WFD | 22.15 | 3.91 | 0.94 | 77.07 | 2.75 | 0.92 |
| WFD w/ KV sharing | 27.27 | 4.81 | 0.97 | - | - | |
| Conv. SD w/ SmolLM2-135M | - | - | - | 72.44 | 2.59 | 0.80 |
As shown in Table F, WFD achieves slightly higher throughput than conventional speculative decoding with SmolLM2-135M on Ouro-2.6B (77.07 vs. 72.44 tok/s). However, given the relatively small difference, we consider their performance comparable rather than claiming that WFD outperforms conventional speculative decoding. For Huginn, which uses its own separately trained tokenizer, no suitable public drafter is available. However, since WFD benefits strongly from a deeper recurrence depth (32 for Huginn vs. 4 for Ouro) and from cross-recurrence KV sharing, we expect WFD to offer greater performance benefits in Huginn-like settings.
Rather than claiming that WFD outperforms conventional speculative decoding at this point, we would like to emphasize the following contributions:
- No additional drafter: WFD achieves throughput comparable to conventional speculative decoding without requiring additional training or a separate draft model. This demonstrates that competitive serving performance can be achieved without the additional model-weight memory and inference overhead associated with an external drafter.
-A new pipeline schedule: WFD introduces a novel wavefront pipeline schedule that overlaps drafting and verification within a single looped model, enabling speculative decoding without a separate drafting phase or external drafter.
We revised the paper to include the additional experiments and clarified claims. We hope we have addressed your concerns and questions regarding the paper. If any concerns remain, we would be happy to discuss them further.
[1] R.-J. Zhu et al., "Scaling Latent Reasoning via Looped Language Models," arXiv:2510.25741, 2025.
[2] V. Conchello Vendrell et al., "Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models," in Proc. ICML, 2026.
[3] J. O'Neill and F. Reid, "Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers," arXiv:2607.15456, 2026.
[4] Y. Chen et al., "Hybrid Latent Attention for Looped Language Models," arXiv:2610.07940, 2026.
[5] Y. Bai et al., "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding," in Proc. ACL, 2024.
[6] L. Ben Allal et al., "SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model," arXiv:2502.02737, 2025.
We sincerely thank you for the insightful comments on our manuscript. As the PCs highly encourage brevity, we provide concise descriptions.
Response to Weakness 1
We appreciate your raising this important question regarding the practical effectiveness of WFD. We address this concern through the experiments in our responses to Questions 2, 3, and 4 below.
Question 2
For Question 2, to distinguish the effects of AR weight amortization and WFD-specific overhead, we provide a per-token latency breakdown at batch size of 1 in Table A, and the component-wise speedups relative to AR at batch sizes 1 and 8 in Table B.
Table A. Per-token latency breakdown.
Table B. Per-token latency Speedup breakdown. Speedup is the per-output-token latency ratio relative to AR.
As shown in Table A, scheduling overhead is negligible for both models. The dominant latency components are attention, tokenwise operations (e.g., GEMM and layer normalization), and kernel launches.
Table B provides further insight into the causes of the speedup reduction as batch size increases. First, the speedup of tokenwise operations remains relatively stable, changing from 4.13$\times$ to 3.95$\times$ for Huginn and from 2.76$\times$ to 3.12$\times$ for Ouro. This suggests that improved weight amortization in the AR baseline is not the dominant factor behind the reduced WFD speedup.
In contrast, the attention speedup decreases from 0.91$\times$ to 0.75$\times$ for Huginn and from 1.44$\times$ to 0.94$\times$ for Ouro, indicating that WFD-specific attention overhead becomes more pronounced at larger batch sizes. This overhead is largely associated with cross-recurrence KV traffic.
The significance of KV traffic is further supported by the results with KV sharing. For Huginn at batch size 8, enabling cross-recurrence KV sharing increases the attention speedup to 4.31$\times$ and the overall speedup to 3.38$\times$. These results demonstrate that eliminating redundant KV traffic can substantially mitigate the batch-size-related performance degradation.
Although full cross-recurrence KV sharing is not applicable to Ouro without accuracy degradation, recent studies [2–4] have explored alternative cross-loop KV sharing and compression techniques. We leave the integration of WFD with such techniques to future work.