- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Depth-Asynchronous Self-Speculation for looped Transformers. The method decouples the recurrent depth of draft computation from the depth of cached verified-prefix representations. Its full decoder, DAS-Wave, combines mature prefix K/V reads with carried parallel refinement and independent full-depth verification. Experiments across four recurrent-model checkpoints and mathematics/code tasks report 4.00–6.96× speedup over full-depth autoregressive decoding.
Critical strengths
-
A useful new design dimension for recurrent self-speculation. Allowing shallow draft computation to access mature verified-prefix representations is a well-motivated idea. The channel-intervention experiments provide evidence that prefix representation depth can substantially influence draft quality without increasing the draft's recurrent computation.
-
Substantial empirical improvements with informative component analyses. DAS-Wave achieves strong speedups across the evaluated recurrent checkpoints. The experiments examine the interaction between mature K/V reads, carried latent states, and proposal computation budgets, providing useful evidence beyond the headline performance numbers.
Critical weaknesses
-
The incremental contribution over existing parallel refinement methods and tuned baselines remains insufficiently isolated. DAS-Wave combines the new depth-asynchronous prefix-read policy with carried parallel refinement and progressive window growth inherited from prior work. Although the paper includes component comparisons and retuned controls, the headline 4–7× speedups are relative to AR and should not be interpreted as gains solely from the newly proposed prefix-read mechanism. A stronger direct comparison with the inherited parallel proposal machinery would clarify the incremental contribution. Moreover, the sequential self-speculative baseline searches draft depths 4 and 8 but excludes depths 1 and 2, even though the paper demonstrates useful shallow predictions at these depths. This may underestimate the performance of cheaper sequential drafting configurations.
-
Limited performance generalization beyond the evaluated workloads and recurrence settings. The experiments are restricted to mathematics and coding benchmarks, models evaluated at recurrence depth 32, and batch-size-one execution. The correctness argument preserves greedy target decoding under its assumptions, so the primary concern is not output quality degradation but whether the acceleration mechanism remains effective under different generation patterns and computation-depth regimes.
Questions for authors
-
Could the authors provide a direct comparison between DAS-Wave and a well-tuned parallel-refinement baseline with conventional prefix reads, using the same inference stack and verifier? The existing component results are useful, but reporting the incremental speedup attributable specifically to depth-asynchronous reads would better establish the new contribution.
-
Could the authors extend the sequential self-speculative baseline search to draft depths 1 and 2, jointly tuning speculative block length? Reporting accepted length, proposal latency, and end-to-end speedup would help determine whether the current baseline underestimates the performance of shallower configurations.
-
Could the authors include one less structured or more context-dependent generation workload, reporting accepted length and speedup? Where supported, a second target recurrence-depth setting would also help establish whether the method's benefits depend on the selected depth of 32.
-
How is the prefix-read depth determined for the carried Aligned control, given that carried latent states do not correspond to a unique ordinary recurrent depth? Clarifying this mapping would strengthen the interpretation of the component ablations.
6 comments
Author Rebuttal (2/3)
Question 3 / Weakness 2: Generalization across generation workloads and target recurrence depths.
Response. Following your suggestion, we added a general-language generation workload and evaluated different target recurrence depths. We report the Huginn results below. Speedups follow the mean paired definition in Eq. (8).
(1) A less structured generation workload. We selected 40 prompts from MT-Bench, with 10 each from the writing, roleplay, extraction, and humanities categories. We evaluated first-turn generation with R=32, batch size 1, greedy decoding, a 512-token generation limit, and three timing repetitions:
| Metric | AR | Lookahead | PLD | Token-Recycling | SAM-Decoding | Self-speculative | Mature-Spec | DAS-Wave |
|---|---|---|---|---|---|---|---|---|
| Throughput (tokens/s) | 12.40 | 15.51 | 14.78 | 28.70 | 15.65 | 18.92 | 24.07 | 48.51 |
| Speedup | 1.00× | 1.31× | 1.27× | 2.35× | 1.38× | 1.53× | 2.00× | 3.87× |
| Accepted length A | — | 0.40 | 0.45 | 1.59 | 0.52 | 4.69 | 4.24 | 14.93 |
DAS-Wave achieves a 3.87× speedup with an average accepted draft length of 14.93 tokens, showing that its efficiency gains extend to these general-language generation prompts beyond mathematics and code.
(2) Different target recurrence depths. We evaluated R∈{16,32,64} on the original GSM8K prompts, with three timing repetitions. Each setting was compared against AR at the same target depth and used the corresponding depth-R verifier. The DAS-Wave proposal configuration selected at R=32 was transferred without retuning:
| Method | Metric | R=16 | R=32 | R=64 |
|---|---|---|---|---|
| AR | Throughput (tokens/s) | 22.90 | 11.97 | 6.21 |
| Speedup | 1.00× | 1.00× | 1.00× | |
| Self-speculative | Throughput (tokens/s) | 24.43 | 20.28 | 14.79 |
| Speedup | 1.06× | 1.68× | 2.39× | |
| Accepted length A | 5.55 | 5.52 | 5.45 | |
| DAS-Wave | Throughput (tokens/s) | 54.40 | 49.72 | 37.45 |
| Speedup | 2.35× | 4.04× | 5.84× | |
| Accepted length A | 15.60 | 17.58 | 17.80 |
DAS-Wave outperforms AR and sequential self-speculation at all three target depths, showing that its acceleration is not limited to R=32. Relative speedups increase with R in these experiments, while absolute throughput decreases.
Question 4: Prefix-read depth in the carried Aligned control.
Response. In the carried Aligned control, update s=1,...,k within each refinement round reads depth-s prefix K/V at the same physical core layer. Each round restarts the read index from 1 while retaining the hidden states. Thus, alignment follows the update within the round, without assuming that the carried state corresponds to an ordinary recurrent depth. The read policies are:
| Policy | Prefix K depth | Prefix V depth |
|---|---|---|
| Aligned | s | s |
| Deep V | s | R |
| Deep K | R | s |
| Deep K/V | R | R |
We use this rule to compare all controls at the same refinement budget without requiring cache entries beyond R. For the parallel controls, the shallow-read rule is the same with and without carry, and the refinement schedule and target verifier are held fixed.
We additionally tested cumulative prefix reads, where the read depth advances across rounds instead of restarting. With R=32, k=2, and J=16, the cumulative indices remain within 1–32 without clipping. Both policies retain carry and use the same refinement schedule and verifier. Results for Huginn on the original GSM8K prompts, with three timing repetitions, are shown below:
| Metric | Cumulative K/V (J=16) | DAS-Wave (J=16) |
|---|---|---|
| Throughput (tokens/s) | 44.65 | 47.63 |
| Speedup | 3.56× | 3.79× |
DAS-Wave achieves higher throughput and mean paired speedup over AR at this matched budget, showing an additional benefit from mature prefix K/V even when the reference reads deepen across rounds.
Author Rebuttal (3/3)
Weakness 2: Performance across batch sizes and sampling temperatures.
Response. We evaluated different batch sizes and sampling temperatures to assess efficiency beyond the original single-request, greedy setting. Batch speedups average paired throughput ratios over batches; sampling speedups follow Eq. (8), with paired ratios averaged over prompts and seeds.
(1) Performance across batch sizes. We evaluated static batching at R=32 with greedy decoding. The table below reports throughput and speedup for Huginn on GSM8K:
| Method | Metric | B=1 | B=2 | B=4 | B=8 | B=16 |
|---|---|---|---|---|---|---|
| AR | Throughput (tokens/s) | 11.64 | 19.01 | 29.49 | 44.88 | 53.28 |
| Speedup | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | |
| DSpark | Throughput (tokens/s) | 39.87 | 49.00 | 58.10 | 74.32 | 79.43 |
| Speedup | 3.44× | 2.48× | 1.85× | 1.52× | 1.38× | |
| DAS-Wave | Throughput (tokens/s) | 53.29 | 74.60 | 101.71 | 125.18 | 130.33 |
| Speedup | 4.46× | 3.74× | 3.21× | 2.55× | 2.32× |
As batch size increases from 1 to 16, both DSpark and DAS-Wave achieve higher throughput, while their speedups over AR decrease. DAS-Wave maintains a throughput advantage over DSpark across all reported batch sizes, showing that its gains extend beyond single-request execution.
(2) Performance under non-greedy decoding. We evaluated T∈{0.3,0.5,0.7} at fixed top-p=0.9 using Huginn and Raven-Llama on the original GSM8K prompts. The experiments use R=32, batch size 1, and three sampling seeds, with proposal configurations kept fixed across temperatures:
| Model | Method | Metric | T=0.3 | T=0.5 | T=0.7 |
|---|---|---|---|---|---|
| Huginn | AR | Throughput (tokens/s) | 13.03 | 13.23 | 12.96 |
| Speedup | 1.00× | 1.00× | 1.00× | ||
| DSpark | Throughput (tokens/s) | 34.74 | 35.12 | 32.53 | |
| Speedup | 2.74× | 2.71× | 2.59× | ||
| DAS-Wave | Throughput (tokens/s) | 44.38 | 39.15 | 31.17 | |
| Speedup | 3.48× | 3.07× | 2.54× | ||
| Raven-Llama | AR | Throughput (tokens/s) | 12.33 | 12.74 | 12.36 |
| Speedup | 1.00× | 1.00× | 1.00× | ||
| DSpark | Throughput (tokens/s) | 35.60 | 35.48 | 32.52 | |
| Speedup | 2.95× | 2.85× | 2.70× | ||
| DAS-Wave | Throughput (tokens/s) | 50.21 | 43.58 | 32.65 | |
| Speedup | 4.16× | 3.50× | 2.77× |
As temperature increases, the speedups of both methods decrease. DAS-Wave still achieves substantial speedups of 2.54–4.16× over AR and provides greater acceleration than DSpark in most of the reported settings. These results support its effectiveness across the evaluated sampling temperatures.
We appreciate the reviewer’s constructive suggestions. The main additions are marked in blue in the revised manuscript. We hope the new experimental results and clarifications address the concerns raised.
Author Rebuttal (1/3)
We thank the reviewer for the thoughtful review and for recognizing the motivation and empirical strength of our work. Below we respond point by point to the concerns raised.
Response. The reported 4.00–6.96× speedups describe the complete DAS-Wave decoder relative to AR. Following your suggestion, we added a comparison with a parallel-refinement baseline using cumulative prefix reads. We independently selected proposal configurations for both methods on development data.
Both methods retain carry and progressive window growth and use the same inference stack and full-depth verifier. The selected configurations use R=32, γ=32, and J=24, with the active window growing by one position per round. For both tasks, Huginn uses k=2, with initial window widths of 4 for the cumulative control and 2 for DAS-Wave; Raven-Llama uses k=1, with initial widths of 1 and 2, respectively. For the cumulative-read baseline, the prefix-cache depth advances across refinement rounds and is capped at R; DAS-Wave reads depth-R prefix K/V throughout.
The following results use the prompt sets from the original evaluation, with three timing repetitions. Throughput is reported in tokens/s, and speedups are mean paired per-prompt ratios relative to AR, following Eq. (8):
DAS-Wave achieves higher measured throughput and mean speedup over AR in all four settings. The largest direct paired throughput gain over the independently tuned baseline is approximately 6.5%, observed on Raven-Llama/GSM8K. To further isolate the contribution of depth-asynchronous prefix reads, we also performed a matched-configuration ablation for Huginn on GSM8K (R=32, k=2, J=16). Replacing cumulative K/V reads with depth-R K/V reads increases the mean speedup over AR from 3.56× to 3.79× (see our response to Question 4 below).
Response. Thank you for this suggestion. We expanded the sequential self-speculative baseline search to d∈{1,2,4,8}, jointly tuning speculative block length over b∈{2,4,8,16,32}. Configurations were selected on development data and fixed before testing.
We evaluated the selected configurations at d=1 and d=2, the original sequential configuration from Table 5, and DAS-Wave on the original GSM8K test prompts using Huginn and Raven-Llama. All methods use the same inference stack and full-depth verifier, with R=32, greedy decoding, and batch size 1. End-to-end measurements use three timing repetitions, and speedups follow Eq. (8). Proposal latency was profiled separately on eight development prompts. The original configurations were also remeasured under these matched conditions:
The expanded search favors d=2 on Raven-Llama, with a modest increase in measured speedup from 3.09× to 3.21×, while Huginn still favors its original d=8 configuration. Shallower drafts reduce proposal latency, but their shorter accepted lengths can offset this saving. Even after jointly tuning draft depth and block length, DAS-Wave achieves substantially higher speedups than the best sequential configurations found: 3.98× versus 1.68× on Huginn and 5.30× versus 3.21× on Raven-Llama.