acceptodds
Under review as a conference paper at ICLR 2027

WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models

Abstract

Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves speedup on Ouro-2.6B and on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to on Huginn-3.5B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient AI, efficient LLM inference, looped language models, recurrent-depth transformers, speculative decoding, self-speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_vhYmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes Wavefront Decoding (WFD), a training-free self-speculative decoding method for looped language models. WFD uses shallow recurrence outputs as drafts and batches token states at different positions and recurrence depths into a diagonal wavefront. This overlaps drafting, recurrent computation, and full-depth verification within the same recurrent-block calls. Rejected drafts are corrected using full-depth predictions. The paper provides a latency model showing that WFD reaches the asymptotic latency of Draft-then-Verify (DtV) with a finite wavefront. Experiments on Ouro-2.6B and Huginn-3.5B report 2.42x and 3.54x speedups over autoregressive decoding and a 1.27x improvement over DtV.

Critical strengths

WFD is a natural and well motivated schedule for looped models. It uses shallow recurrence outputs as drafts while exploiting weight sharing to batch states at different recurrence depths. This removes the phase boundary in DtV without requiring another model or additional training. The construction applies to both full-stack and P/R/C looped architectures.

Critical weaknesses
  1. The evidence for realistic serving workloads or at larger model scales is limited. (1) The main experiments use two 2.6B/3.5B models on a single A6000, mostly at user batch size 1. The appendix shows that speedup decreases as batch size grows, particularly without cross-recurrence KV sharing which is model-dependent, approximate and could not be applied to Ouro without harming accuracy. (2) Fixed-batch experiments also omit several effects that interact directly with the proposed schedule, such as continuous batching, heterogeneous output lengths, and rejection-induced wavefront flushing. These effects can reduce the effective recurrent batch size or introduce bubbles and queueing between the P/R/C stages. (3) No larger or distributed target model is evaluated, so it is unclear whether mixed-depth recurrent calls remain nearly as cheap as single-position calls once execution becomes compute- or communication-bound. Since WFD is primarily a serving schedule, these missing evidence substantially limit the practical conclusions.
  2. The evaluation lacks conventional speculative decoding with an external draft model as baselines. The paper presents the lack of auxiliary training and draft-model weights as an advantage, but only compares WFD with AR and the self-speculative DtV baseline. It therefore remains unclear whether WFD offers a favorable throughput-memory trade-off against a small external drafter, or whether its training-free design sacrifices attainable speed. A comparison under similar memory and quality constraints is needed to assess the practical value of this design choice.
Questions for authors
  1. Was an external draft model considered or tested with either target model? If so, how did its throughput, memory use, and acceptance rate compare with WFD? If not, is there a technical reason this baseline is unsuitable for looped targets?
  2. In the batch-size experiments, how much of the speedup reduction comes from better weight amortization in the AR baseline, and how much comes from WFD-specific KV traffic or increased recurrent-block latency?
  3. Can the authors provide a continuous-batching experiment using variable prompt and output lengths, preferably reporting throughput and p95/p99 latency?
  4. Can the authors provide results on a larger or distributed looped model, or profiling evidence showing that mixed-depth recurrent calls remain bandwidth-bound as model size and effective batch size increase?
9 comments

We sincerely thank you for the insightful comments on our manuscript. As the PCs highly encourage brevity, we provide concise descriptions.

Response to Weakness 1

We appreciate your raising this important question regarding the practical effectiveness of WFD. We address this concern through the experiments in our responses to Questions 2, 3, and 4 below.

Question 2

For Question 2, to distinguish the effects of AR weight amortization and WFD-specific overhead, we provide a per-token latency breakdown at batch size of 1 in Table A, and the component-wise speedups relative to AR at batch sizes 1 and 8 in Table B.

Table A. Per-token latency breakdown.

ComponentHuginn-3.5BOuro-2.6B
Attention27.5%14.2%
Tokenwise ops57.9%22.1%
Kernel launch14.2%63.3%
Scheduling0.2%0.1%
Miscellaneous0.3%0.3%
total100%100%

Table B. Per-token latency Speedup breakdown. Speedup is the per-output-token latency ratio relative to AR.

Batch sizeModelModeOverall Speedup vs. ARComponent-wise Speedup vs. AR
AttentionTokenwise opsKernel launch
1Huginn-3.5BWFD3.44$\times$0.91$\times$4.13$\times$5.54$\times$
WFD w/ KV sharing3.47$\times$4.12$\times$4.11$\times$2.27$\times$
Ouro-2.6BWFD2.59$\times$1.44$\times$2.76$\times$2.79$\times$
8Huginn-3.5BWFD1.46$\times$0.75$\times$3.95$\times$1.74$\times$
WFD w/ KV sharing3.38$\times$4.31$\times$3.95$\times$1.82$\times$
Ouro-2.6BWFD1.93$\times$0.94$\times$3.12$\times$2.90$\times$

As shown in Table A, scheduling overhead is negligible for both models. The dominant latency components are attention, tokenwise operations (e.g., GEMM and layer normalization), and kernel launches.

Table B provides further insight into the causes of the speedup reduction as batch size increases. First, the speedup of tokenwise operations remains relatively stable, changing from 4.13$\times$ to 3.95$\times$ for Huginn and from 2.76$\times$ to 3.12$\times$ for Ouro. This suggests that improved weight amortization in the AR baseline is not the dominant factor behind the reduced WFD speedup.

In contrast, the attention speedup decreases from 0.91$\times$ to 0.75$\times$ for Huginn and from 1.44$\times$ to 0.94$\times$ for Ouro, indicating that WFD-specific attention overhead becomes more pronounced at larger batch sizes. This overhead is largely associated with cross-recurrence KV traffic.

The significance of KV traffic is further supported by the results with KV sharing. For Huginn at batch size 8, enabling cross-recurrence KV sharing increases the attention speedup to 4.31$\times$ and the overall speedup to 3.38$\times$. These results demonstrate that eliminating redundant KV traffic can substantially mitigate the batch-size-related performance degradation.

Although full cross-recurrence KV sharing is not applicable to Ouro without accuracy degradation, recent studies [2–4] have explored alternative cross-loop KV sharing and compression techniques. We leave the integration of WFD with such techniques to future work.

Question 3

To the best of our knowledge, none of the existing serving engines supports Huginn. We therefore built a custom serving engine that supports full CUDA-graph execution with continuous batching, natural rejection and scheduling.

To assess how WFD performs under more realistic serving conditions (variable prompt and output lengths), we evaluated it on LongBench [5] with the custom engine at a concurrency of 8.

Table C. Serving performance of Ouro-2.6B-Thinking on 280 LongBench samples at a concurrency of 8. Prompt lengths range from 2048 to 16384 tokens (mean 7431, std 5365).

MetricARDtVWFD
tok/s (×\times AR)20.7225.15 (1.21$\times$)26.99 (1.30$\times$)
Decode tok/s (×\times AR)22.6928.04 (1.24$\times$)30.44 (1.34$\times$)
α–0.770.77
TPOT p95/p99 (ms)114.2/120.0108.5/122.898.7/106.6
E2E latency p95/p99 (s)207.6/230.0174.2/210.8154.0/172.3
Effective concurrency2.02.02.2

Table D. Serving performance of Huginn-3.5B on 100 LongBench samples at a concurrency of 8. Prompt lengths range from 193 to 2048 tokens (mean 1598, std 420).

MetricARDtVWFD w/ KV sharing
tok/s (×\times AR)19.1230.11 (1.57$\times$)42.17 (2.21$\times$)
Decode tok/s (×\times AR)28.3460.48 (2.13$\times$)139.91 (4.94$\times$)
α–0.910.97
TPOT p95/p99 (ms)339/352326/365245/399
E2E latency p95/p99 (s)59.5/63.740.5/46.832.4/40.9
Effective concurrency6.06.07.6

Table C shows that WFD outperforms DtV on 2K–16K-token contexts at a concurrency of 8. Huginn was trained with (and defaults to) a 4096-token context length, so we evaluated it on contexts of up to 2K tokens in Table D. On Ouro, WFD shows a lower speedup than in our main results, but still outperforms DtV. The reduced gain mainly stems from the lower acceptance rate on LongBench and longer contexts. On Huginn, WFD achieves 4.94$\times$ decode speedup with KV sharing, demonstrating its effectiveness in a serving setting. WFD also shows lower E2E p95/p99 latency on both models.

Question 4

We acknowledge your concerns about scalability to larger models and batch sizes. To the best of our knowledge, no larger publicly available checkpoints exist for either model type (full-stack and P/R/C). Ouro-2.6B and Huginn-3.5B have hidden dimensions of 2048 and 5280, and loop-unrolled parameter counts of ~10B and ~54B, respectively. These models are therefore comparable in per-token compute and weight traffic to dense models with tens of billions of parameters, such as Llama-2-13B or Qwen3-32B.
To assess wavefront decoding’s scalability to larger hidden dimensions, we instantiated dummy-weight models with dimensions adapted from Kimi K2 (hidden dimension of 7168, 56 query heads, 4:1 GQA, head dimension of 128, FFN dimension of 18432, vocabulary size of 163840), following the Ouro (full-stack) and Huginn (P/R/C) architectures.

Table E. Speedup of each decoding method on dummy-weight models (batch 1, 1024-token
prefill, 512 decode steps, fixed at the overall rates from the Spec-Bench results).

ARDtVWFD
Dummy-Huginn-Big1.00$\times$2.89$\times$4.28$\times$
Dummy-Ouro-Big1.00$\times$1.79$\times$3.04$\times$

Table E shows the speedup of WFD on dummy-weight models with fixed acceptance rates. WFD achieves higher speedups than in our main results because the larger hidden dimension and GQA ratio shift the memory-traffic bottleneck from attention to the FFN, which favors WFD even without cross-recurrence KV sharing.

We also provide an arithmetic-intensity analysis in Appendix C.1. In short, with or without cross-recurrence KV sharing, the decoding phase remains memory-bandwidth-bound for practical batch sizes (up to ~200).

Response to Weakness 2, Question 1

We sincerely appreciate the insightful question. As you pointed out, requiring neither additional training nor a separate drafter is an advantage of WFD. Nevertheless, we agree that comparing WFD against conventional speculative decoding (SD) is important for assessing its practical serving performance.

To the best of our knowledge, no external draft model has been specifically designed or trained for Ouro or Huginn. To provide an initial comparison, we conducted additional experiments using SmolLM2-135M [6], a publicly available model that shares Ouro's tokenizer, as the drafter (Table F). We selected SmolLM2-135M because it outperformed SmolLM2-360M in our experiments. For Huginn, which uses its own separately trained tokenizer, no suitable public drafter is available.

Table F. Throughput on the custom serving engine across Spec-Bench tasks (concurrency = 1). α denotes the acceptance rate. All other settings follow the main results.

ModeHuginn-3.5BOuro-2.6B
Throughput (tok/s)SpeedupAlphaThroughput (tok/s)SpeedupAlpha
vLLM AR---27.46--
AR5.671.0028.011.00
DtV15.352.710.9352.091.860.91
WFD22.153.910.9477.072.750.92
WFD w/ KV sharing27.274.810.97--
Conv. SD w/ SmolLM2-135M---72.442.590.80

As shown in Table F, WFD achieves slightly higher throughput than conventional speculative decoding with SmolLM2-135M on Ouro-2.6B (77.07 vs. 72.44 tok/s). However, given the relatively small difference, we consider their performance comparable rather than claiming that WFD outperforms conventional speculative decoding. For Huginn, which uses its own separately trained tokenizer, no suitable public drafter is available. However, since WFD benefits strongly from a deeper recurrence depth (32 for Huginn vs. 4 for Ouro) and from cross-recurrence KV sharing, we expect WFD to offer greater performance benefits in Huginn-like settings.
Rather than claiming that WFD outperforms conventional speculative decoding at this point, we would like to emphasize the following contributions:

- No additional drafter: WFD achieves throughput comparable to conventional speculative decoding without requiring additional training or a separate draft model. This demonstrates that competitive serving performance can be achieved without the additional model-weight memory and inference overhead associated with an external drafter.

-A new pipeline schedule: WFD introduces a novel wavefront pipeline schedule that overlaps drafting and verification within a single looped model, enabling speculative decoding without a separate drafting phase or external drafter.

We revised the paper to include the additional experiments and clarified claims. We hope we have addressed your concerns and questions regarding the paper. If any concerns remain, we would be happy to discuss them further.

[1] R.-J. Zhu et al., "Scaling Latent Reasoning via Looped Language Models," arXiv:2510.25741, 2025.
[2] V. Conchello Vendrell et al., "Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models," in Proc. ICML, 2026.
[3] J. O'Neill and F. Reid, "Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers," arXiv:2607.15456, 2026.
[4] Y. Chen et al., "Hybrid Latent Attention for Looped Language Models," arXiv:2610.07940, 2026.
[5] Y. Bai et al., "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding," in Proc. ACL, 2024.
[6] L. Ben Allal et al., "SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model," arXiv:2502.02737, 2025.

Reviewer_CPKPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WFD, a self-speculative decoding method for looped language models. It batches tokens at different recurrence depths, allowing shallow drafting to overlap with full-depth verification. Incorrect drafts trigger correction and removal of dependent speculative states. Experiments on Ouro and Huginn report average speedups of 2.42× and 3.54× over autoregressive decoding, respectively, and approximately 27% over draft-then-verify. Additional experiments examine approximate KV-cache sharing and the effects of acceptance rate, batch size, and context length.

Critical strengths

The paper offers a practical improvement to inference in looped models. Its strongest evidence combines consistent speedups with direct checks of decoding correctness. The systems analysis also helps explain the observed gains.

  1. Batching token states at different recurrence depths makes effective use of shared weights without requiring an additional draft model or retraining.
  2. Full-depth verification and removal of dependent speculative states provide a clear mechanism for preserving greedy decoding while reusing partial computation.
  3. The main experiments show consistent improvements across six task categories and two model families, including approximately 27% higher throughput than draft-then-verify.
  4. The reported fp32 checks achieve complete token agreement with autoregressive decoding for both models, strengthening the correctness claim beyond task-level accuracy comparisons.
  5. The profiling and ablations connect performance to acceptance rates, draft depth, kernel overhead, and cache traffic, making the contribution informative beyond the headline speedups.
Critical weaknesses

My main concern is how much of the reported gain survives a fully optimised comparison. Ouro’s autoregressive baseline spends about 65% of token latency in launch or idle overhead; the CUDA-graph results are encouraging but cover a static steady state. A matched, end-to-end comparison of autoregressive decoding, draft-then-verify, and WFD should account for dynamic rejection, refill, and sequence termination.

The scaling claims also need tighter boundaries. The batch-size and context-length sweeps force acceptance rates, and WFD with separate recurrence caches eventually falls behind draft-then-verify in some settings (Figures 5–6). Cache sharing improves performance but changes the computation, with accuracy checks limited to the reported reasoning benchmarks. In particular, the shared-cache timing curves for Ouro need clearer qualification because the paper reports that this configuration fails to preserve accuracy.

Finally, the claim of compatibility with arbitrary sampling is broader than the demonstrated algorithm. Algorithm 1 and the exact-match experiments establish greedy behaviour; stochastic decoding additionally requires probability-based acceptance and residual correction. The paper should specify these operations and justify their compatibility with the scheduling and cache invariants, or narrow its correctness claims to the evaluated setting.

Questions for authors

my first Q is that how much of WFD’s advantage remains when all three decoders receive comparable execution optimizations in complete runs with natural rejection and refill?

the second Q: Which larger-batch and longer-context configurations retain their speed advantage under naturally occurring acceptance rates? What quality evidence supports the shared-cache configurations shown in those plots?

Third one is that what acceptance and residual-sampling procedure supports the general sampling claim, and has it been implemented or validated?

lastly, what distinguishes the exact scheduling mechanism from LoopSpec’s closely related pipelined self-speculation? Its September 2026 release makes it concurrent work; clarifying the relationship would help establish the contribution’s precise scope.

3 comments

We sincerely thank you for the insightful comments on our manuscript. As the PCs highly encourage brevity, we provide concise descriptions.

Response to Weakness 1, Question 1

We sincerely appreciate your thorough examination of our experimental setup and the important question. To the best of our knowledge, none of the existing serving engines supports Huginn. To validate WFD's advantage under execution optimizations comparable to those of production serving engines, we built a custom serving engine that supports full CUDA-graph execution with natural rejection and refill.

Table A. Throughput on the custom serving engine across Spec-Bench tasks (concurrency = 1). denotes the acceptance rate. All other settings follow Table 1.

ModeHuginn-3.5BOuro-2.6B
Throughput (tok/s)Speedupα\alphaThroughput (tok/s)SpeedupAlpha
vLLM AR---27.46--
AR5.671.00×28.011.00×
DtV15.352.71×0.9352.091.86×0.91
WFD22.153.91×0.9477.072.75×0.92
WFD w/ KV sharing27.274.81×0.97--

Table A reports optimized end-to-end generation throughput on Spec-Bench, with the same experimental setup as Table 1 in Section 4. The optimized engine increases the AR throughput of Ouro-2.6B from 6.62 to 28.01 tok/s, comparable to that of vLLM (27.46 tok/s). It also increases the AR throughput of Huginn-3.5B from 5.29 to 5.67 tok/s. In the optimized engine, WFD achieves even higher speedup than in the eager setup (Table 1 in Section 4), from 3.54× (2.42×) to 3.91× (2.75×) on Huginn (Ouro). The cross-recurrence sharing in Huginn-3.5B further increases the speedup to 4.81× by reducing the KV memory traffic.

Response to Weakness 2, Question 2

We thank you for the opportunity to clarify the relationship between the cross-recurrence KV sharing and the scalability to batch size and context length.We first clarify the rationale for the shared-cache configuration. The Huginn paper [1] reports that KV-cache sharing has minimal impact on performance, with MT-Bench scores statistically indistinguishable from the baseline. Our results on GSM8K and MATH-500 also indicate that the approximation introduced by KV sharing has a limited effect on task accuracy. We therefore view KV sharing on Huginn models as a natural extension. In contrast, the Ouro paper [2] reports that KV sharing preserves accuracy only when applied during the decoding phase. Nevertheless, cross-loop KV sharing and compression methods for Ouro have been actively studied in recent work [3–5]. We leave combining WFD with such methods on full-stack architectures (e.g., Ouro) to future work.

Cross-recurrence KV traffic is the sole bottleneck to scaling WFD to long context. Figure 6 in Appendix B shows that the speedup of WFD is maintained with KV sharing. On the other hand, scaling WFD to large batch sizes is hindered by two factors, the cross-recurrence KV traffic and memory-compute balance shift. As speculative decoding exploits the memory-bandwidth-bound nature of the decoding phase, its performance gains typically diminish at large batch sizes, a trend also observed in conventional speculative decoding for autoregressive (AR) models.

To assess how WFD performs without KV sharing under more realistic serving conditions, we evaluated it on LongBench with the custom engine at a concurrency of 8.

Table B. Serving performance of Ouro-2.6B-Thinking on 280 LongBench samples at a concurrency of 8. Prompt lengths range from 2048 to 16384 tokens (mean 7431, std 5365).

Ouro-2.6B-thinkingARDtVWFD
tok/s (× AR)20.7225.15 (1.21×)26.99 (1.30×)
Decode tok/s (× AR)22.6928.04 (1.24×)30.44 (1.34×)
α–0.770.77
TPOT p95/p99 (ms)114.2/120.0108.5/122.898.7/106.6
E2E latency p95/p99 (s)207.6/230.0174.2/210.8154.0/172.3
Effective concurrency2.02.02.2

Table C. Serving performance of Huginn-3.5B on 100 LongBench samples at a concurrency of 8. Prompt lengths range from 193 to 2048 tokens (mean 1598, std 420).

Huginn-3.5BARDtVWFD w/ KV sharing
tok/s (× AR)19.1230.11 (1.57×)42.17 (2.21×)
Decode tok/s (× AR)28.3460.48 (2.13×)139.91 (4.94×)
α–0.910.97
TPOT p95/p99 (ms)339/352326/365245/399
E2E latency p95/p99 (s)59.5/63.740.5/46.832.4/40.9
Effective concurrency6.06.07.6

Table B shows that WFD outperforms DtV on 2K–16K-token contexts at a concurrency of 8. Huginn was trained with (and defaults to) a 4096-token context length, so we evaluated it with contexts of up to 2K tokens in Table C. On Ouro, WFD shows a lower speedup than in our main results, but still outperforms DtV. The reduced gain mainly stems from lower acceptance on LongBench and longer contexts. On Huginn, WFD achieves 4.94× decode speedup with KV sharing, showing effectiveness under a serving environment.

Response to Weakness 3, Question 3

We acknowledge your concerns about non-greedy sampling. We note that stochastic sampling via speculative sampling [6, 7], i.e., rejection sampling with residual resampling, can be applied to WFD without additional scheduling cost. We evaluated WFD under speculative sampling, which preserves the target distribution, at temperatures of 0.7 and 1.0.

Table D. WFD under speculative sampling (100 randomly sampled problems from MATH-500).

Ouro-2.6B-Thinking
TemperatureTPS (tok/s)SpeedupAcceptance rateAccuracy (%)
Greedy (T=0)15.322.37×0.9259
T=0.715.172.35×0.9259
T=1.015.292.36×0.9156

In Table D, WFD maintains its acceptance rate and speedup at T = 0.7 and 1.0. Greedy sampling corresponds to speculative decoding at T=0, where it preserves the target’s greedy (one-hot) distribution. We used Ouro-2.6B-Thinking because non-thinking models frequently failed to emit the EOS token properly after producing the final answer, and subsequently exhibited degenerate generation even under AR decoding.

Response to Question 4

We acknowledge your concern regarding the novelty of WFD relative to LoopSpec, and appreciate the opportunity to clarify the relationship between the two works. As you pointed out, LoopSpec is a concurrent work developed independently of WFD. Upon close examination, we think that wavefront decoding and LoopSpec share core mechanisms, namely shallow-depth draft proposal and recurrent-block pipelining. WFD and the single-proposal pipeline of LoopSpec are conceptually the same algorithm, and the difference in their reported speedups likely stems from differences in hardware and workload setups.
We view the distinct contributions of WFD and LoopSpec as follows:

  • LoopSpec introduces a second draft proposal to further improve the acceptance rate of self-speculative decoding.
  • Our work analyzes the speedup of the proposed decoding in terms of KV memory traffic, and extends it to long-context and large-batch workloads via cross-recurrence KV sharing.

We revised the paper to include the additional experiments and clarified claims. We hope we have addressed your concerns and questions regarding the paper. If any concerns remain, we would be happy to discuss them further.

[1] J. Geiping et al., "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach," in Proc. NeurIPS, 2025.
[2] R.-J. Zhu et al., "Scaling Latent Reasoning via Looped Language Models," arXiv:2510.25741, 2025.
[3] V. Conchello Vendrell et al., "Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models," in Proc. ICML, 2026.
[4] J. O'Neill and F. Reid, "Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers," arXiv:2607.15456, 2026.
[5] Y. Chen et al., "Hybrid Latent Attention for Looped Language Models," arXiv:2610.07940, 2026.
[6] Y. Leviathan, M. Kalman, and Y. Matias, "Fast Inference from Transformers via Speculative Decoding," in Proc. ICML, 2023.
[7] C. Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling," arXiv:2302.01318, 2023.

Reviewer_NDDJOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Wavefront Decoding (WFD), a training-free self-speculative decoding framework tailored for looped language models. Looped architectures increase effective depth by repeatedly applying a weight-shared block, but the sequential recurrent calls per token substantially increase decoding latency. WFD exploits this weight sharing by processing token states at different positions and recurrence depths simultaneously in a single batched call, forming a "diagonal wavefront". This design fuses the traditionally separate drafting and verification phases into a continuous pipeline, where rejected drafts are corrected using full-depth predictions. Experiments show WFD achieves 2.42× speedup on Ouro-2.6B and 3.54× speedup on Huginn-3.5B over autoregressive (AR) decoding, scaling up to 4.81× when combined with cross-recurrence KV sharing.

Critical strengths
  1. Unlike standard Draft-then-Verify (DtV) methods that suffer from phase separation, WFD concurrently batches drafting and verification steps, eliminating phase boundaries and maximizing the utilization of shared weights in looped models.

  2. WFD requires no additional training. The authors also released the code and experimental scripts, providing reasonably good reproducibility.

Critical weaknesses
  1. It lacks comparison to LoopSpec, which provides a similar method.

  2. The lossless sampling claim is unsupported. All experiments use greedy decoding. Under stochastic sampling, simply comparing a draft token with a full-depth sampled token and replacing it upon disagreement does not automatically preserve the target distribution.

Questions for authors
  1. What are the algorithmic differences between WFD and LoopSpec’s single-proposal pipeline? Can the authors provide a direct comparison using the same hardware and checkpoints?

  2. For non-greedy sampling, what acceptance probability and residual sampling distribution are used? Is there a proof of distribution preservation?

1 comment

We sincerely thank you for the insightful comments on our manuscript. As the PCs highly encourage brevity, we provide concise descriptions.

Response to Weakness 1, Question 1

We acknowledge your concern regarding the novelty of WFD relative to LoopSpec, and appreciate the opportunity to clarify the relationship between the two works. LoopSpec was released on arXiv on September 15, 2026, which makes it concurrent work developed independently of WFD (WFD was first submitted on September 16). Upon close examination, we think that wavefront decoding and LoopSpec share core mechanisms, namely shallow-depth draft proposal and recurrent-block pipelining. WFD and the single-proposal pipeline of LoopSpec are conceptually the same algorithm, and the difference in their reported speedups likely stems from differences in hardware and workload setups.
We view the distinct contributions of WFD and LoopSpec as follows:

  • LoopSpec introduces a second draft proposal to further improve the acceptance rate of self-speculative decoding.
  • Our work analyzes the speedup of the proposed decoding in terms of KV memory traffic, and extends it to long-context and large-batch workloads via cross-recurrence KV sharing.

Response to Weakness 2, Question 2

We acknowledge your concerns about non-greedy sampling. We note that stochastic sampling via speculative sampling [1] [2], i.e., rejection sampling with residual resampling, can be applied to WFD without additional scheduling cost. We evaluated WFD under speculative sampling, which preserves the target distribution, at temperatures of 0.7 and 1.0.

Table A. WFD under speculative sampling (100 randomly sampled problems from MATH-500).

Ouro-2.6B-Thinking
TemperatureTPS (tok/s)SpeedupAcceptance rateAccuracy (%)
Greedy (T=0)15.322.37×0.9259
T=0.715.172.35×0.9259
T=1.015.292.36×0.9156

In Table A, WFD maintains its acceptance rate and speedup at T = 0.7 and 1.0. Greedy sampling corresponds to speculative decoding at T=0, where it preserves the target’s greedy (one-hot) distribution. We used Ouro-2.6B-thinking, because non-thinking models frequently failed to emit the EOS token properly after producing the final answer, and subsequently exhibited degenerate generation even under AR decoding.

We revised the paper to include the additional experiments. We hope we have addressed your concerns and questions regarding the paper. If any concerns remain, we would be happy to discuss them further.

[1] Y. Leviathan, M. Kalman, and Y. Matias, "Fast Inference from Transformers via Speculative Decoding," in Proc. ICML, 2023.
[2] C. Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling," arXiv:2302.01318, 2023.

Reviewer_nMdNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WaveFront Decoding (WFD), a training-free self-speculative decoding schedule for looped language models, whose repeated sequential applications of a shared block cause high decoding latency. WFD batches token states at different positions and recurrence depths in the same shared-block invocation, continuously overlapping shallow drafting with full-depth verification. Upon rejection, it commits the full-depth prediction and flushes subsequent speculative states. Across six Spec-Bench categories, WFD achieves 2.42× and 3.54× speedups over autoregressive decoding on Ouro-2.6B and Huginn-3.5B, respectively, and approximately 1.27× over Draft-then-Verify (DtV) on both models. Cross-recurrence KV sharing raises the maximum Huginn speedup to 4.81×, although this is an approximate extension.

Critical strengths
  1. Unlike phase-separated DtV, WFD batches drafting and verification in the same recurrent-block call, reducing serial invocations and amortizing weight reads. It requires neither an auxiliary draft model nor additional training and applies to both full-stack and P/R/C architectures.

  2. Consistent improvements across the six categories, controlled acceptance-rate experiments, and analyses of batch size, context length, and draft depth help explain the interaction among parallelism, rejection recovery, and KV traffic.

  3. Controlled experiments with fixed acceptance rates compare WFD and DtV across batch sizes and analyze why their relative speedups decline as concurrency increases. Larger batches already amortize weight reads across more requests, while additional speculative computation and KV traffic further limit the remaining gains. Comparing configurations with and without KV sharing also demonstrates attention to the trade-off between resource overhead and speculative benefit at higher concurrency.

Critical weaknesses
  1. The dataset size and workload coverage are limited, particularly for code generation. The six Spec-Bench categories belong to one compact evaluation suite and should not be equated with six large, independent application benchmarks. The original Spec-Bench selects 80 instances per category, whereas this paper does not explicitly report the actual number used in each category, making sensitivity to sample selection difficult to assess. Conversation, translation, summarization, question answering, mathematical reasoning, and RAG also do not adequately establish performance on the output structures and verification behavior of program generation. Although the paper additionally evaluates mathematical accuracy on GSM8K and MATH-500, those experiments do not replace dedicated code-generation or code-completion evaluation. The authors should disclose sample counts, broaden the evaluation where feasible, and add code workloads with acceptance rates, end-to-end speedups, and task correctness.

  2. The evaluation is restricted to approximately 3B-parameter models, leaving scalability to larger models unestablished. Experiments cover only Ouro-2.6B and Huginn-3.5B, without comparisons on larger looped models. Changes in shared-block size, hidden dimension, and model depth can alter the balance among weight reads, KV traffic, batching costs, and shallow-to-full-depth prediction agreement, so the reported speedups cannot be directly extrapolated to larger models. The authors state that they use the largest public checkpoints from the two model families; this explains part of the experimental limitation but does not establish generalization across scales. A representative larger looped checkpoint would strengthen the evaluation if one is available. Otherwise, the authors should explicitly scope their claims and provide cost or performance analysis across block sizes, without needing to train a new model from scratch.

  3. Acceleration in multi-turn agentic interactions and realistic long-context generation remains insufficiently validated. The main experiments use prompts of at most 1,024 tokens and cap outputs at 512 tokens. Although the appendix evaluates different context lengths and batch sizes, these controlled timing experiments with fixed acceptance rates do not fully capture actual performance when tool interactions progressively extend the context, tool outputs accumulate, and draft acceptance varies across turns and content. The authors should add representative multi-turn tool-use trajectories and compare AR, DtV, and WFD across turn positions and context lengths. Reporting per-request latency and cumulative model inference time over complete sessions, with prefill and decoding costs separated, would establish the practical speedup on agentic workloads.

Questions for authors

See the weakness

1 comment

We sincerely thank you for the insightful comments on our manuscript. As the PCs highly encourage brevity, we provide concise descriptions.

Response to Weakness 1

We thank you for pointing out the ambiguity of the experiment setup. Our main throughput experiments used the same 80 instances per category as the original Spec-Bench. For code-generation or code-completion evaluation, we conducted additional experiments on HumanEval [1].

Table A. Performances on the HumanEval dataset (concurrency 1).

TaskMetricOuro-2.6BHuginn-3.5B
ARDtVWFD (ours)ARDtVWFD (ours)
HumanEvalα-0.930.93-0.910.92
Speedup1.001.942.781.002.473.51
Pass@10.3960.3780.3840.2010.1950.201

Table A reports end-to-end speedup on HumanEval, which consists of 164 code-completion problems. The acceptance rates range from 0.91 to 0.93, consistent with the Spec-Bench results. Consequently, WFD achieves a 2.78× speedup over AR on Ouro and 3.51× on Huginn, both of which are also consistent with the Spec-Bench results.

Response to Weakness 2

We acknowledge your concerns about scalability to larger models. To the best of our knowledge, no larger publicly available checkpoints exist for either model type (full-stack and P/R/C). Ouro-2.6B and Huginn-3.5B have hidden dimensions of 2048 and 5280, and loop-unrolled parameter counts of ~10B and ~54B, respectively. Those models are therefore comparable in per-token compute and weight traffic to dense models with tens of billions of parameters, such as Llama-2-13B or Qwen3-32B. To assess wavefront decoding’s scalability to larger hidden dimensions, we instantiated dummy-weight models with dimensions adapted from Kimi K2 (hidden dimension of 7168, 56 query heads, 4:1 GQA, head dimension of 128, FFN dimension of 18432, vocabulary size of 163840), following the Ouro (full-stack) and Huginn (P/R/C) architectures.

Table B. Speedup of each decoding method on dummy-weight models (batch 1, 1024-token prefill, 512 decode steps, fixed at the overall rates from the Spec-Bench results).

ARDtVWFD
Dummy-Huginn-Big1.00×2.89×4.28×
Dummy-Ouro-Big1.00×1.79×3.04×

Table B shows the performance of wavefront decoding on dummy-weight models with a fixed acceptance rate. WFD achieves higher speedups than in our main results because the larger hidden dimension and GQA ratio shift the memory-traffic bottleneck from attention to the FFN, which favors WFD even without cross-recurrence KV sharing.

Regarding the shared-block configuration, Ouro and Huginn have substantially different (P, R × loops, C) shapes: (0, 48 × 4, 0) and (2, 4 × 32, 2), respectively. WFD applies to both configurations without modification. Figure 7 in Appendix B also reports Huginn's speedup with varying draft depth Td, showing that WFD achieves similar speedups for Td from 4 to 8.

The remaining question is whether shallow-to-full-depth prediction agreement holds at scale, which cannot be verified without larger pretrained checkpoints. Nevertheless, Blayney et al. [2] show that, in looped models, latent states commonly converge toward a cyclic fixed point along the recurrence. We therefore expect shallow-to-full-depth prediction agreement to persist in larger models.

Response to Weakness 3

To evaluate WFD on agentic, long-context workloads, we report end-to-end generation throughput of Ouro-2.6B on 150 samples of SWE-agent-trajectories [3] in Table C. The samples comprise 50 trajectories each with prefill lengths of <3K, 3K–8K, and 8K–16K tokens. Huginn was trained with (and defaults to) a 4096-token context length, which is insufficient for agentic or long-context workloads.

Table C. The speedups of each decoding on SWE-agent-trajectory samples.

Overall tok/s (× AR)Decode tok/s (× AR)Acceptance rate
AR11.617.3-
DtV16.3 (1.40×)29.7 (1.71×)0.87
WFD18.3 (1.57×)37.1 (2.14×)0.87

Ouro supports a 64K context length, but the Ouro paper does not report training on agentic data. Even so, Ouro produced well-formed outputs in 97% of cases. WFD achieves a higher speedup than DtV.

We revised the paper to include the additional experiments and clarified claims. We hope we have addressed your concerns and questions regarding the paper. If any concerns remain, we would be happy to discuss them further.

[1] M. Chen et al., "Evaluating Large Language Models Trained on Code," arXiv:2107.03374, 2021.
[2] H. Blayney et al., "A Mechanistic Analysis of Looped Reasoning Language Models," arXiv:2604.11791, 2026.
[3] K. Zainullina et al., "Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents," in Proc. ICML, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.