acceptodds
Under review as a conference paper at ICLR 2027

Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Abstract

Looped Transformers reuse a shared block across recurrent depths, making autoregressive decoding expensive because every generated token requires many sequential recurrent passes. Self-speculative decoders reduce this cost by drafting at an early depth and verifying at full depth, but typically bind draft computation to prefix representations from the same recurrent depth. We find that queries and keys approach their final-depth representations earlier than values, and controlled prefix-channel interventions show that mature values substantially improve shallow draft predictions. Motivated by this asymmetry, we introduce **Depth-Asynchronous Self-Speculation (DAS)**, which decouples the depth of draft computation from the depth of verified-prefix representations it reads. Its **Mature-V** primitive lets shallow queries retrieve full-depth prefix values without additional recurrent computation. We further develop **DAS-Wave**, which combines depth-asynchronous prefix reads with carried parallel refinement, progressive block growth, and an independent full-depth verifier. Across four recurrent-model checkpoints and mathematics and code workloads, DAS-Wave achieves **4.00–6.96×** mean throughput speedup over paired full-depth autoregressive decoding in the same inference stack. These results identify prefix-information depth as an effective design axis for recurrent self-speculation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Looped Transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xFCrOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Depth-Asynchronous Self-Speculation for looped Transformers. The method decouples the recurrent depth of draft computation from the depth of cached verified-prefix representations. Its full decoder, DAS-Wave, combines mature prefix K/V reads with carried parallel refinement and independent full-depth verification. Experiments across four recurrent-model checkpoints and mathematics/code tasks report 4.00–6.96× speedup over full-depth autoregressive decoding.

Critical strengths
  1. A useful new design dimension for recurrent self-speculation. Allowing shallow draft computation to access mature verified-prefix representations is a well-motivated idea. The channel-intervention experiments provide evidence that prefix representation depth can substantially influence draft quality without increasing the draft's recurrent computation.

  2. Substantial empirical improvements with informative component analyses. DAS-Wave achieves strong speedups across the evaluated recurrent checkpoints. The experiments examine the interaction between mature K/V reads, carried latent states, and proposal computation budgets, providing useful evidence beyond the headline performance numbers.

Critical weaknesses
  1. The incremental contribution over existing parallel refinement methods and tuned baselines remains insufficiently isolated. DAS-Wave combines the new depth-asynchronous prefix-read policy with carried parallel refinement and progressive window growth inherited from prior work. Although the paper includes component comparisons and retuned controls, the headline 4–7× speedups are relative to AR and should not be interpreted as gains solely from the newly proposed prefix-read mechanism. A stronger direct comparison with the inherited parallel proposal machinery would clarify the incremental contribution. Moreover, the sequential self-speculative baseline searches draft depths 4 and 8 but excludes depths 1 and 2, even though the paper demonstrates useful shallow predictions at these depths. This may underestimate the performance of cheaper sequential drafting configurations.

  2. Limited performance generalization beyond the evaluated workloads and recurrence settings. The experiments are restricted to mathematics and coding benchmarks, models evaluated at recurrence depth 32, and batch-size-one execution. The correctness argument preserves greedy target decoding under its assumptions, so the primary concern is not output quality degradation but whether the acceleration mechanism remains effective under different generation patterns and computation-depth regimes.

Questions for authors
  1. Could the authors provide a direct comparison between DAS-Wave and a well-tuned parallel-refinement baseline with conventional prefix reads, using the same inference stack and verifier? The existing component results are useful, but reporting the incremental speedup attributable specifically to depth-asynchronous reads would better establish the new contribution.

  2. Could the authors extend the sequential self-speculative baseline search to draft depths 1 and 2, jointly tuning speculative block length? Reporting accepted length, proposal latency, and end-to-end speedup would help determine whether the current baseline underestimates the performance of shallower configurations.

  3. Could the authors include one less structured or more context-dependent generation workload, reporting accepted length and speedup? Where supported, a second target recurrence-depth setting would also help establish whether the method's benefits depend on the selected depth of 32.

  4. How is the prefix-read depth determined for the carried Aligned control, given that carried latent states do not correspond to a unique ordinary recurrent depth? Clarifying this mapping would strengthen the interpretation of the component ablations.

6 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We thank the reviewer for the thoughtful review and for recognizing the motivation and empirical strength of our work. Below we respond point by point to the concerns raised.


Question 1 / Weakness 1: Direct comparison with a tuned parallel-refinement baseline.

Response. The reported 4.00–6.96× speedups describe the complete DAS-Wave decoder relative to AR. Following your suggestion, we added a comparison with a parallel-refinement baseline using cumulative prefix reads. We independently selected proposal configurations for both methods on development data.

Both methods retain carry and progressive window growth and use the same inference stack and full-depth verifier. The selected configurations use R=32, γ=32, and J=24, with the active window growing by one position per round. For both tasks, Huginn uses k=2, with initial window widths of 4 for the cumulative control and 2 for DAS-Wave; Raven-Llama uses k=1, with initial widths of 1 and 2, respectively. For the cumulative-read baseline, the prefix-cache depth advances across refinement rounds and is capped at R; DAS-Wave reads depth-R prefix K/V throughout.

The following results use the prompt sets from the original evaluation, with three timing repetitions. Throughput is reported in tokens/s, and speedups are mean paired per-prompt ratios relative to AR, following Eq. (8):

ModelTaskAR TPSCumulative control: TPS / SpeedupDAS-Wave: TPS / Speedup
HuginnGSM8K11.9748.35 / 3.96×49.72 / 4.04×
HuginnMBPP12.5356.50 / 3.93×58.79 / 4.09×
Raven-LlamaGSM8K11.3155.69 / 5.02×59.19 / 5.31×
Raven-LlamaMBPP11.7872.41 / 5.98×74.95 / 6.17×

DAS-Wave achieves higher measured throughput and mean speedup over AR in all four settings. The largest direct paired throughput gain over the independently tuned baseline is approximately 6.5%, observed on Raven-Llama/GSM8K. To further isolate the contribution of depth-asynchronous prefix reads, we also performed a matched-configuration ablation for Huginn on GSM8K (R=32, k=2, J=16). Replacing cumulative K/V reads with depth-R K/V reads increases the mean speedup over AR from 3.56× to 3.79× (see our response to Question 4 below).


Question 2 / Weakness 1: Extending the sequential self-speculative baseline search.

Response. Thank you for this suggestion. We expanded the sequential self-speculative baseline search to d∈{1,2,4,8}, jointly tuning speculative block length over b∈{2,4,8,16,32}. Configurations were selected on development data and fixed before testing.

We evaluated the selected configurations at d=1 and d=2, the original sequential configuration from Table 5, and DAS-Wave on the original GSM8K test prompts using Huginn and Raven-Llama. All methods use the same inference stack and full-depth verifier, with R=32, greedy decoding, and batch size 1. End-to-end measurements use three timing repetitions, and speedups follow Eq. (8). Proposal latency was profiled separately on eight development prompts. The original configurations were also remeasured under these matched conditions:

ModelMethodDraft configurationAccepted length AProposal latency (ms/block)Speedup
HuginnSelf-speculative (original)d=8, b=85.52160.001.68×
HuginnSelf-speculatived=1, b=40.5620.061.11×
HuginnSelf-speculatived=2, b=41.2928.601.40×
HuginnDAS-WaveOriginal configuration17.58192.863.98×
Raven-LlamaSelf-speculative (original)d=4, b=1612.73209.493.09×
Raven-LlamaSelf-speculatived=1, b=165.7596.942.61×
Raven-LlamaSelf-speculatived=2, b=169.59134.433.21×
Raven-LlamaDAS-WaveOriginal configuration18.98163.985.30×

The expanded search favors d=2 on Raven-Llama, with a modest increase in measured speedup from 3.09× to 3.21×, while Huginn still favors its original d=8 configuration. Shallower drafts reduce proposal latency, but their shorter accepted lengths can offset this saving. Even after jointly tuning draft depth and block length, DAS-Wave achieves substantially higher speedups than the best sequential configurations found: 3.98× versus 1.68× on Huginn and 5.30× versus 3.21× on Raven-Llama.

AuthorsOpenReview ↗

Author Rebuttal (2/3)

Question 3 / Weakness 2: Generalization across generation workloads and target recurrence depths.

Response. Following your suggestion, we added a general-language generation workload and evaluated different target recurrence depths. We report the Huginn results below. Speedups follow the mean paired definition in Eq. (8).

(1) A less structured generation workload. We selected 40 prompts from MT-Bench, with 10 each from the writing, roleplay, extraction, and humanities categories. We evaluated first-turn generation with R=32, batch size 1, greedy decoding, a 512-token generation limit, and three timing repetitions:

MetricARLookaheadPLDToken-RecyclingSAM-DecodingSelf-speculativeMature-SpecDAS-Wave
Throughput (tokens/s)12.4015.5114.7828.7015.6518.9224.0748.51
Speedup1.00×1.31×1.27×2.35×1.38×1.53×2.00×3.87×
Accepted length A—0.400.451.590.524.694.2414.93

DAS-Wave achieves a 3.87× speedup with an average accepted draft length of 14.93 tokens, showing that its efficiency gains extend to these general-language generation prompts beyond mathematics and code.

(2) Different target recurrence depths. We evaluated R∈{16,32,64} on the original GSM8K prompts, with three timing repetitions. Each setting was compared against AR at the same target depth and used the corresponding depth-R verifier. The DAS-Wave proposal configuration selected at R=32 was transferred without retuning:

MethodMetricR=16R=32R=64
ARThroughput (tokens/s)22.9011.976.21
Speedup1.00×1.00×1.00×
Self-speculativeThroughput (tokens/s)24.4320.2814.79
Speedup1.06×1.68×2.39×
Accepted length A5.555.525.45
DAS-WaveThroughput (tokens/s)54.4049.7237.45
Speedup2.35×4.04×5.84×
Accepted length A15.6017.5817.80

DAS-Wave outperforms AR and sequential self-speculation at all three target depths, showing that its acceleration is not limited to R=32. Relative speedups increase with R in these experiments, while absolute throughput decreases.


Question 4: Prefix-read depth in the carried Aligned control.

Response. In the carried Aligned control, update s=1,...,k within each refinement round reads depth-s prefix K/V at the same physical core layer. Each round restarts the read index from 1 while retaining the hidden states. Thus, alignment follows the update within the round, without assuming that the carried state corresponds to an ordinary recurrent depth. The read policies are:

PolicyPrefix K depthPrefix V depth
Alignedss
Deep VsR
Deep KRs
Deep K/VRR

We use this rule to compare all controls at the same refinement budget without requiring cache entries beyond R. For the parallel controls, the shallow-read rule is the same with and without carry, and the refinement schedule and target verifier are held fixed.

We additionally tested cumulative prefix reads, where the read depth advances across rounds instead of restarting. With R=32, k=2, and J=16, the cumulative indices remain within 1–32 without clipping. Both policies retain carry and use the same refinement schedule and verifier. Results for Huginn on the original GSM8K prompts, with three timing repetitions, are shown below:

MetricCumulative K/V (J=16)DAS-Wave (J=16)
Throughput (tokens/s)44.6547.63
Speedup3.56×3.79×

DAS-Wave achieves higher throughput and mean paired speedup over AR at this matched budget, showing an additional benefit from mature prefix K/V even when the reference reads deepen across rounds.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

Weakness 2: Performance across batch sizes and sampling temperatures.

Response. We evaluated different batch sizes and sampling temperatures to assess efficiency beyond the original single-request, greedy setting. Batch speedups average paired throughput ratios over batches; sampling speedups follow Eq. (8), with paired ratios averaged over prompts and seeds.

(1) Performance across batch sizes. We evaluated static batching at R=32 with greedy decoding. The table below reports throughput and speedup for Huginn on GSM8K:

MethodMetricB=1B=2B=4B=8B=16
ARThroughput (tokens/s)11.6419.0129.4944.8853.28
Speedup1.00×1.00×1.00×1.00×1.00×
DSparkThroughput (tokens/s)39.8749.0058.1074.3279.43
Speedup3.44×2.48×1.85×1.52×1.38×
DAS-WaveThroughput (tokens/s)53.2974.60101.71125.18130.33
Speedup4.46×3.74×3.21×2.55×2.32×

As batch size increases from 1 to 16, both DSpark and DAS-Wave achieve higher throughput, while their speedups over AR decrease. DAS-Wave maintains a throughput advantage over DSpark across all reported batch sizes, showing that its gains extend beyond single-request execution.

(2) Performance under non-greedy decoding. We evaluated T∈{0.3,0.5,0.7} at fixed top-p=0.9 using Huginn and Raven-Llama on the original GSM8K prompts. The experiments use R=32, batch size 1, and three sampling seeds, with proposal configurations kept fixed across temperatures:

ModelMethodMetricT=0.3T=0.5T=0.7
HuginnARThroughput (tokens/s)13.0313.2312.96
Speedup1.00×1.00×1.00×
DSparkThroughput (tokens/s)34.7435.1232.53
Speedup2.74×2.71×2.59×
DAS-WaveThroughput (tokens/s)44.3839.1531.17
Speedup3.48×3.07×2.54×
Raven-LlamaARThroughput (tokens/s)12.3312.7412.36
Speedup1.00×1.00×1.00×
DSparkThroughput (tokens/s)35.6035.4832.52
Speedup2.95×2.85×2.70×
DAS-WaveThroughput (tokens/s)50.2143.5832.65
Speedup4.16×3.50×2.77×

As temperature increases, the speedups of both methods decrease. DAS-Wave still achieves substantial speedups of 2.54–4.16× over AR and provides greater acceleration than DSpark in most of the reported settings. These results support its effectiveness across the evaluated sampling temperatures.


We appreciate the reviewer’s constructive suggestions. The main additions are marked in blue in the revised manuscript. We hope the new experimental results and clarifications address the concerns raised.

Reviewer_gyM1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper lets shallow draft computation in looped Transformers use deeper representations of the verified prefix. DAS-Wave combines this with state reuse and parallel updates across rounds, followed by full-depth verification.

Critical strengths
  1. The ablations examine the roles of keys, values, and state reuse separately, providing useful support for the speedup results.

  2. The method improves speed across several models and tasks while retaining full-depth verification to preserve greedy decoding.

Critical weaknesses
  1. I am still unsure how much the prefix-read policy differs from existing cache-sharing methods. Once the prefix has been fully computed, what is new about reading its deeper cached representations?

  2. With states carried across rounds, the choice of cache depth for the controls is not obvious. This needs to be clearer to understand what the ablations show.

  3. The earlier analysis emphasizes mature values, but mature keys become much more important with state reuse. The connection between these findings needs more explanation.

Questions for authors
  1. For a fully computed prefix, how do deep K/V reads differ in practice from cache sharing in Huginn and Parallel Samplers?

  2. How is the shallow cache depth chosen in the carried-state controls, and how are updates across rounds handled?

  3. Why do mature keys help much more with state reuse? How does this relate to the earlier analysis and the updates across rounds?

4 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We sincerely thank the reviewer for the detailed feedback and constructive questions. Below we respond point by point to the concerns raised.

Weakness / Question 1: Comparison with earlier cache policies.

Response. We would like to clarify the relationship between our method and the cache policies used in Huginn and Parallel Samplers. Although the methods can share the same local prefix-read operation when selecting the same depth-R K/V, their complete decoding procedures differ in how these representations are obtained and used. We explain these similarities and differences below.

(1) Relationship to Huginn's cache policies. Huginn's standard AR implementation uses depth-aligned K/V by default, with last-pass K/V sharing available as an optional policy. Even when enabled, generation remains sequential: each new token still requires all R recurrent updates before the next token is generated. This policy alone therefore does not introduce speculative acceleration. DAS-Wave uses mature prefix K/V by default for shallow drafting and verifies multiple proposed tokens together, while retaining the standard full-depth target computation.

(2) Relationship to Parallel Samplers. Parallel Samplers also shares K/V across recurrent depths. Its official implementation defaults to adaptive freezing and retains each position's latest K/V, which need not match the standard depth-R target representations. Under both fixed and adaptive freezing, tokens are committed without independent target verification, so agreement with the target's greedy decoding is not guaranteed. DAS-Wave uses depth-R K/V from the verified prefix for drafting and independently verifies proposals with the unchanged full-depth target, preserving its greedy decoding rule.

(3) Experimental comparison. We compared the four decoding configurations on the same GSM8K prompts using Huginn with our SGLang adaptation, on the same hardware with batch size 1 and three timing repetitions. Both AR variants and the DAS-Wave verifier use R=32. Speedups follow the mean paired per-prompt definition in Eq. (8).

MetricHuginn ARLast-pass KV ARParallel SamplersDAS-Wave
Throughput (tokens/s)21.1221.2369.3874.46
Speedup1.00×1.01×3.30×3.52×
Answer accuracy (%)25.0025.0022.5025.00

Last-pass K/V reuse alone provides little throughput improvement in this full-depth AR configuration. DAS-Wave achieves the highest throughput among these configurations while matching AR's observed answer accuracy on this subset.


Weakness / Question 2: Prefix-cache indexing in the carried controls.

Response. In the carried Aligned and single-channel controls in Tables 2 and 9, the shallow prefix-cache depth follows the update index within the current refinement round, rather than accumulated updates across rounds. Specifically, let s=1,...,k denote the update within a round. At the same physical core layer, the read policies are:

PolicyPrefix K depthPrefix V depth
Alignedss
Deep VsR
Deep KRs
Deep K/VRR

For Huginn with k=2, shallow channels read depths 1 and 2 in every round. In Table 9, a K/V channel indicator of 0 selects this round-local shallow representation, while 1 selects depth R. We chose this rule to compare all controls at the same refinement budget without requiring cache entries beyond R. The shallow-read rule is identical with and without carry, and the drafting schedule and target verifier are held fixed. Carry preserves hidden states across rounds without changing this indexing rule. Thus, no cache indices exceed R, and no clipping or wraparound is needed.

Motivated by your question, we additionally evaluated cumulative reads, whose indices advance across refinement rounds. Cumulative reads would require cache entries beyond R if Jk>R. Using the carried parallel setup in Table 2, we therefore compared three policies at R=32, k=2, and J=16, keeping all cumulative indices within R without clipping. The indices follow 1,2 / 3,4 / ... / 31,32. All three configurations use the same refinement schedule and verifier. The table below reports throughput and mean paired speedups for Huginn on the original GSM8K prompts, based on three timing repetitions. We also include a separately measured DAS-Wave configuration with J=24:

MetricCumulative K/V (J=16)Deep V, cumulative K (J=16)DAS-Wave (J=16)DAS-Wave (J=24)
Throughput (tokens/s)44.6546.5847.6348.06
Speedup3.56×3.71×3.79×4.03×

DAS-Wave achieves higher throughput and speedup than both cumulative controls at the matched J=16 budget. At J=24, its prefix-cache index remains fixed at R, requiring no deeper prefix caches, and the measured speedup further increases to 4.03× over AR.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

Weakness / Question 3: The role of mature keys under carry.

Response. We would like to clarify that Figure 2 primarily aims to demonstrate the benefit of mature prefix V for shallow drafting. Its channel interventions therefore use a simple, controlled setting with draft depth d=1, without parallel refinement or carry. In this setting, Deep V outperforms Deep K/V on Huginn, whereas Deep K/V performs better than Deep V on Raven-Llama.

Table 2 provides a more complete evaluation through component ablations of the decoder, including parallel refinement with carry. Under these carried drafting conditions, adding mature K to mature V provides an additional speedup on both Huginn and Raven-Llama, demonstrating its contribution to the full decoder. These findings show that the effect of mature K varies across drafting conditions. Below, we analyze the roles of the shallow-key reference and draft depth in explaining this difference.

(1) The shallow-key reference matters. As clarified in our response to Question 2, the original controls with carry retain hidden states across rounds but restart shallow cache reads at each round. The queries therefore continue to evolve while the prefix keys are repeatedly read from the first few recurrent depths. This may affect query–key compatibility and contribute to the large benefit of mature K in the original round-local comparison (1.07× to 4.11× at J=24).

Our additional cumulative controls allow the reference keys to deepen across rounds. At the matched J=16 budget, with carry and mature V fixed, replacing cumulative K with mature K increases the mean speedup over AR from 3.71× to 3.79×. The gain is smaller in this additional comparison, but mature K remains beneficial even when the reference keys deepen across rounds.

(2) The effect of mature K depends on the model and draft depth. Beyond the choice of shallow-key reference, Table 6 shows that the effect of mature K also changes with draft depth. For Huginn, the draft–target agreement rates are:

Prefix policyd=1d=2
Aligned47.22%63.62%
Deep V66.93%76.59%
Deep K46.83%60.45%
Deep K/V60.05%79.37%

With mature V fixed, adding mature K reduces agreement by 6.88 percentage points at d=1 but improves it by 2.78 points at d=2. One possible explanation is that additional recurrent updates make the queries more compatible with mature prefix keys. These results help explain why the depth-1 preference for Deep V in Figure 2 need not extend to Huginn's parallel drafting setting in Table 2, which uses k=2 recurrent updates per refinement round. The effect also depends on the model. Figure 2 shows that Raven-Llama's Q/K representations are already closer to their final-depth counterparts at the first recurrence than those of Huginn. For Raven-Llama, Deep K/V already outperforms Deep V at d=1 (90.21% versus 87.96%). These results indicate that the benefit of mature K depends on both the model and draft depth.

Regarding §4.5, input-conditioned recurrence allows carried states to incorporate revised token guesses across rounds, giving them a computation history different from the ordinary target trajectory in Figure 2. This difference may also play a role, but our interpretation of the observed gains is primarily based on the comparisons of prefix reads and draft depth above.


We thank the reviewer again for the thoughtful questions. We have highlighted the main additions to the revised manuscript in blue. We hope our explanations and supporting evidence address the reviewer’s concerns.

Thank you for the additional experiments and discussion; I have increased my score.

Thank you for your follow-up and for taking the time to consider our additional experiments and clarifications. We appreciate your constructive feedback, which has helped us improve the paper. Please let us know if you have any remaining concerns or questions; we would be happy to address them.

Reviewer_NGWwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether shallow drafters in looped Transformers can improve draft quality and decoding efficiency by accessing deeper representations of verified-prefix keys and values. It introduces Depth-Asynchronous Self-Speculation (DAS) and Mature-V, motivated by the observation that queries and keys mature earlier than values across recurrent depth, and combines these ideas with carried parallel refinement in DAS-Wave. Experiments across four recurrent-model checkpoints and four mathematics and code benchmarks show that DAS-Wave achieves 4.00–6.96× throughput speedups over full-depth autoregressive decoding, outperforming the evaluated training-free decoding baselines in all tested settings. Channel-wise interventions and component ablations provide evidence that mature prefix representations improve draft acceptance, with the combination of mature K/V reads and latent-state carry yielding particularly strong gains.

Critical strengths

The paper provides systematic channel-wise interventions and component ablations that connect prefix K/V depth to draft quality and decoding efficiency, with consistent benefits across the evaluated recurrent checkpoints.

Critical weaknesses

Limited evaluation of practical efficiency. Experiments use batch size 1 and report relative speedups without absolute throughput (tokens/s). They also omit comparisons with trained drafters such as DFlash, DSpark, and EAGLE-3. These are important practical baselines given their use in serving systems and demonstrated throughput gains under batched workloads. Additional drafter training can be amortized over large inference volumes and may be worthwhile for better latency or throughput. Being training-free therefore does not, by itself, justify excluding these baselines. Comparisons across batch sizes, with absolute throughput and training costs reported, would clarify the method's practical competitiveness.

Open-source frameworks such as Speculators, TorchSpec, and SpecForge provide a starting point for training these baselines. The contemporaneous LoopSpec (https://arxiv.org/pdf/2609.17184) study also includes a comparison with DFlash.

Questions for authors

Are the observed Q/K/V maturation asymmetry and resulting gains general properties of compatible looped Transformers, or properties of the particular architecture/training combinations evaluated?

5 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We sincerely thank the reviewer for the detailed and constructive feedback, which has helped us improve our work. Following your suggestions, we provide additional experimental results and clarifications below.


Weakness 1: Absolute efficiency and batch size.

Response. We have added absolute throughput measurements and evaluated performance across batch sizes. We report mean paired throughput ratios, averaged over prompts for single-request experiments as in Eq. (8), and over batches for static batching. Added baselines in the original implementation use the saved AR records from the original evaluation; SGLang and batch experiments use their corresponding AR references.

(1) Absolute throughput. We have added throughput results for the methods evaluated in the original paper, using our original implementation. To further assess the practical efficiency of our approach, we also adapted it to SGLang and report both throughput and speedup. Results for the Huginn model on GSM8K under both implementations are summarized below:

ImplementationMetricARLookaheadPLDToken-RecyclingSAM-DecodingSelf-speculativeMature-SpecDAS-Wave
OriginalThroughput (tokens/s)11.0020.5516.9929.4018.6118.4225.1146.69
Speedup1.00×1.83×1.54×2.81×1.63×1.70×2.26×4.12×
SGLangThroughput (tokens/s)21.3228.3330.4042.6732.3432.0041.3874.84
Speedup1.00×1.33×1.39×1.97×1.48×1.49×1.95×3.50×

DAS-Wave achieves strong speedups in our original implementation and reaches 74.84 tokens/s with a 3.50× speedup over AR under SGLang. These results show that its efficiency gains persist with an optimized inference backend. We have added results for additional models to the appendix.

(2) Performance across batch sizes. We have also added experiments across batch sizes. The table below reports throughput and speedup for the Huginn model on GSM8K:

MethodMetricB=1B=2B=4B=8B=16
ARThroughput (tokens/s)11.6419.0129.4944.8853.28
Speedup1.00×1.00×1.00×1.00×1.00×
DSparkThroughput (tokens/s)39.8749.0058.1074.3279.43
Speedup3.44×2.48×1.85×1.52×1.38×
DAS-WaveThroughput (tokens/s)53.2974.60101.71125.18130.33
Speedup4.46×3.74×3.21×2.55×2.32×

As batch size increases from 1 to 16, both DSpark and DAS-Wave achieve higher throughput, while their speedups over AR decrease. DAS-Wave maintains a throughput advantage over DSpark across all evaluated batch sizes, supporting its effectiveness under batched workloads.


Weakness 2: Comparisons with trained drafters.

Response. We trained target-specific DFlash, DSpark, and EAGLE-3 drafters for all four original target models using the public DeepSpec implementations. The combined cost of data preparation and training across the four targets averaged approximately 30 GPU-hours per drafter family. DAS-Wave requires no additional drafter training. To broaden the comparison, we also evaluated the complete LoopSpec method. All methods were evaluated on the same hardware. For the Huginn model on GSM8K, we obtain the following results:

ImplementationMetricARDFlashDSparkEAGLE-3LoopSpecDAS-Wave
OriginalThroughput (tokens/s)11.0034.3038.5139.2817.6646.69
Speedup1.00×3.15×3.51×3.60×1.62×4.12×
SGLangThroughput (tokens/s)21.3243.5851.9850.5551.6474.84
Speedup1.00×2.08×2.45×2.38×2.37×3.50×

DAS-Wave outperforms the evaluated trained drafters in both the original implementation and the SGLang adaptation, without requiring additional drafter training. Under SGLang, it achieves approximately 44% higher throughput than DSpark, demonstrating its practical competitiveness in this setting.

AuthorsOpenReview ↗

Author Rebuttal (2/2)

Question: Generality of the observation and method.

Response. Our additional experiments extend the evidence for the Q/K/V maturation pattern beyond the original checkpoints. As discussed in §4.5, DAS-Wave uses input-conditioned recurrence to update carried latent states when token guesses change, a form of parallel latent refinement also studied in Parallel Samplers [1]. Our main experiments demonstrate strong speedups in this setting.

We further examined Ouro-1.4B, which uses a different recurrent architecture: it repeats a stack of 24 Transformer layers four times without re-injecting token embeddings at each recurrence. Following the analysis in Figure 2(a,b), the table below reports layer-averaged Q/K/V cosine similarities to the fourth recurrence along the ordinary target trajectory:

RecurrenceQKV
10.8440.8600.621
20.9620.9690.859
30.9890.9920.960

Consistent with the observations in our original paper, Q and K approach their final-recurrence representations earlier than V on average.

For Ouro, we made model-specific adjustments to DAS-Wave, such as resetting draft states at each refinement round. We compared this adaptation with AR and sequential self-speculation on GSM8K using Ouro-1.4B and Ouro-2.6B, with the target depth fixed at R=4:

ModelMetricARSelf-speculativeDAS-Wave
Ouro-1.4BThroughput (tokens/s)20.9736.0839.71
Speedup1.00×1.75×1.91×
Ouro-2.6BThroughput (tokens/s)10.3320.3921.85
Speedup1.00×2.00×2.11×

The adapted method achieves 1.91× and 2.11× speedups over AR and outperforms sequential self-speculation on both models. These results show that our approach can be adapted to another looped architecture.


We again thank the reviewer for the constructive feedback. The main additions to the revised manuscript are highlighted in blue. We hope these additional experiments and clarifications address the concerns raised.


References
[1] "Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models", ICML 2026.

Dear Reviewer NGWw,

With the discussion phase ending in about a day, we would greatly appreciate any further guidance or confirmation. We wanted to follow up on our previous response to your valuable suggestions. Your insights are very important to us, and we are eager to ensure that we have addressed all concerns appropriately.

Warm regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Dear Reviewers,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate any further thoughts on our rebuttal and additional experiments. We hope the new evidence and clarifications have addressed your concerns. If any questions remain or any points would benefit from further explanation, we would be happy to clarify them and continue the discussion.

Best regards,

The Authors

AuthorsOpenReview ↗

Rebuttal Summary

We thank the Area Chair and reviewers for their thoughtful and constructive feedback. We appreciate the reviewers’ positive comments on the design motivation for separating draft depth from prefix-read depth (xFCr), the systematic channel interventions and component ablations linking prefix depth to draft quality and decoding efficiency (NGWw), and the speed improvements across models and tasks that DAS-Wave achieves while retaining full-depth verification (gyM1). We have responded point by point to all three reviews and answered the follow-up questions. The main additions to the revised manuscript are highlighted in blue. We summarize the updates and contributions below.

  1. The contribution of mature prefix reads (xFCr, gyM1). We revised Section 4.3 to distinguish the overall speedup of DAS-Wave from the contribution of its prefix-read policy. Table 1 reports mean paired speedups of 3.87–6.96× for the complete decoder over full-depth autoregressive decoding (AR). In the controlled comparison on Huginn/GSM8K, both runs used carry and the same 16-round refinement schedule and verifier. Switching from cumulative to full-depth prefix K/V reads raised the mean paired speedup over AR from 3.56× to 3.79× (Appendix G, Table 21). In a separate comparison where DAS-Wave and the cumulative-read baseline were tuned independently, DAS-Wave achieved higher measured throughput in all four model–task settings. On Huginn/GSM8K, DAS-Wave independently verified each proposed block with the full-depth target and achieved higher throughput than both AR with last-pass K/V sharing and Parallel Samplers.

  2. Practical efficiency and comparisons with trained drafters (NGWw, xFCr). We added absolute end-to-end throughput measurements for both our original implementation and our SGLang adaptation. We trained target-specific DFlash, DSpark, and EAGLE-3 drafters for each of the four original checkpoints, reported their training costs, and also evaluated LoopSpec. On Huginn/GSM8K in SGLang, DAS-Wave reached 74.84 tokens/s, the highest throughput in this comparison. We also expanded the sequential baseline search to include draft depths 1 and 2 and jointly tuned block length. DAS-Wave remained faster than the best sequential configurations tested on Huginn and Raven-Llama.

  3. Evaluation across workloads, execution settings, and architectures (NGWw, xFCr). We added first-turn MT-Bench prompts covering writing, roleplay, extraction, and humanities to the main evaluation. DAS-Wave retained speedups over AR on this workload, at target recurrence depths of 16, 32, and 64, and under non-greedy sampling. In static batching on Huginn/GSM8K, its throughput exceeded DSpark’s at every tested batch size. In our follow-up, we also reported answer accuracy for DAS-Wave, DSpark, and AR at each batch size on the same 40 prompts. On Ouro-1.4B, queries and keys again approached their final representations earlier than values. A version of DAS-Wave that resets draft states each round achieved speedups over AR on both Ouro-1.4B and Ouro-2.6B, showing that the approach can be adapted to a different looped architecture.

Summary of contributions. DAS separates draft computation depth from the depth of the verified-prefix representations it reads. Its Mature-V primitive lets shallow queries reuse full-depth prefix values without extra recurrent draft updates. DAS-Wave combines mature prefix reads with carried parallel refinement, progressive window growth, and independent full-depth verification. The complete decoder accelerated generation across four recurrent checkpoints and five workloads without training a separate drafter.

We thank the reviewers and the Area Chair again for the suggestions that helped us clarify the contribution of the prefix-read policy and strengthen the evaluation of the complete decoder.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.