acceptodds
Under review as a conference paper at ICLR 2027

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

11 views

Abstract

Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive computational overhead, we introduce a selective second proposal from a deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show that the theoretical predictions closely match empirical measurements. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83 inference speedup across diverse Looped Transformers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Self-Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SF7NOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoopSpec, a method to decode specific steps within a looped transformer and compare with the actual decoded results, to decide whether a branch is accepted or pruned. This way, the recurrent steps within a loop transformer are less as some branches are initiated exactly as the original final decoded token, thus the decoding latency is reduced. The author also designed a gating mechanism to control the size of the branching tree.

The experiments demonstrates the efficiency gain of the proposed method. I would have scored higher. However, the variance is not reported, and it is not clear in the paper what is the cost of expanding a tree in a looped transformer.

Critical strengths
  • The idea and the proposed method is reasonable and very interesting to me.
  • The efficiency gain is clear and substantial.
  • Experiment details are provided, along with code for reproduction.
Critical weaknesses
  • The scope of the experiments is somewhat limited. Only two transformers are compared in this paper.
  • The cost of decoding branches is not declared.
  • The rationale for choosing the proposal depths in Table 1 is not given.
  • The variance is not reported in any experiment。
Questions for authors
  • Can you explain why no variation is provided?
  • I didn't quite get the gating mechanism, could you further explain this? I think adding a bit illustration in Fig. 1 would help, instead of just stating "gate closed, no second proposal"
  • Could you further elaborate the concept a power-law upper envelope ϕ(r)\phi(r) in Fig. 3(b)?
4 comments
AuthorsOpenReview ↗

Initial Response (1/2)

We appreciate reviewer's insightful feedback. We would like to clarify and explain the questions and provide additional experiment results.

Q1/W4. Variance of the Speedup.

We address the reviewer’s concern on missing variance of speedup by the additional experiment in the following table. We ran the Ouro Thinking models under temperature=1.0 and top-p=0.7 and 10 random seeds. The results show consistent speedup over the vanilla models.

ModelMethodGSM8KAIME25
Ouro-1.4B-Thinking1,22.40 ± 0.03x2.34 ± 0.06x
Ouro-2.6B-Thinking1,22.81 ± 0.02x2.59 ± 0.07x

As shown in the table, LoopSpec consistently achieves substantial speedups, with low standard deviations (0.02–0.07x) across random seeds. These results demonstrate that LoopSpec's acceleration is stable and is not sensitive to the choice of random seed.

Q2. Explanation of Gating Mechanism

Overall, we introduce the gating mechanism to tackle the exponentially growing speculative branches in LoopSpec. Intuitively, the gate asks whether the deeper recurrent state still trusts the first draft x~(1)\tilde{x}^{(1)}. When a branch reaches depth d2d_2, we get a second prediction pd2p_{d_2} about the same token x~(1)\tilde{x}^{(1)} that was drafted at depth d1d_1. We compare how much probability each depth assigns to the first draft x~(1)\tilde{x}^{(1)}:

  • Gate closed if pd2(x~(1))≥pd1(x~(1))p_{d_2}(\tilde{x}^{(1)}) \geq p_{d_1}(\tilde{x}^{(1)}): the deeper state is at least as confident, so the first draft is likely to be accepted and a fallback would mostly waste compute.
  • Gate open if pd2(x~(1))<pd1(x~(1))p_{d_2}(\tilde{x}^{(1)}) < p_{d_1}(\tilde{x}^{(1)}): the deeper state has lost confidence, so the first draft is at risk. We then spawn a fallback branch with a second draft.

Without gating, every branch forks at both depths and the number of branches grows exponentially (up to 249 for R=32R = 32, d1=2d_1 = 2, d2=8d_2 = 8). With gating, the gate opens for only 19-40% of branches (Table 8), which reduces the mean effective batch size from 43.75 to 34.06 (Figure 4).

Q3. Explanation of Power-Law Upper Envelope

Figure 3(b) measures how far the intermediate prediction at depth rr is from the final prediction at depth RR, using total variation (TV) distance. This distance empirically decreases as rr grows. To reflect this in our analysis (Theorem 2), we find a simple formula that upper bounds every measured point. In particular, we choose the power law function ϕ(r)=β⋅r−α\phi(r) = \beta \cdot r^{-\alpha}, which appears as a straight line on the log-log plot. The envelope is the tightest line that stays above all measured points: we pick α\alpha and β\beta to minimize the maximum ratio between ϕ(r)\phi(r) and the measured TV.

W1. Limited models are compared

We would like to clarify that in the paper, we conducted experiments across 2 model families, 4 Transformer architectures, and 7 model checkpoints (Table 6), rather than only two Transformers.

To further strengthen the experiment, we additionally provide NVIDIA Nemotron-Labs Dory1.0-4B [1], a recently released Looped Transformer based on normalized-GPT (nGPT) with hybrid sliding-window attention and full attention, as well as GQA, differing from Ouro and Raven's Transformer architectures. We use R=4R=4, and (d1,d2)=(1,2)(d_1, d_2)=(1,2) for Dory.

MethodGSM8KAIME25
Greedy2.29x2.21x
Sampling (T=1, top-p=0.7)2.22x2.19x

As the table shows, LoopSpec performs substantial speedup on the Dory model, demonstrating its capability for more Transformer architectures.

[1] M. Fishman, S. Sun, S. Acharya, F. Ladhak, B. Chmiel, S. A. Serano, R. Banner, B. Ginsburg, and K. C. Puvvada. Dory1.0-4B: A Looped Normalized-GPT. arXiv:2611.03998, 2026.

AuthorsOpenReview ↗

Initial Response (2/2)

W2. The cost of decoding branches is not declared

To quantify the computational overhead of decoding branches, we profile the latency of a single recurrent-block of Raven-Llama under LoopSpec (d1=2d_1=2, d2=8d_2=8) with a 1024-token context. Each branch size is measured 10 times and we report the mean and std.

Branches1246183568130249
Latency per recurrent (μs)696.2 ± 0.5703.4 ± 0.8723.5 ± 0.8755.3 ± 0.5773.5 ± 0.4840.8 ± 0.5972.3 ± 0.31459.3 ± 1.11909.8 ± 1.4
Relative to 1 branch1.00x1.01x1.04x1.09x1.11x1.21x1.40x2.10x2.74x

Branch size 1 to 249 covers the full range of the number of branches in LoopSpec for Raven-Llama model. As shown in the table, the latency increases substantially slower than the number of branches.

Furthermore, the branch size distribution (Figure 4) shows that the average branch size is 34.06 and mostly under 60. Therefore, the additional computation from branching introduces moderate latency overhead due to a memory-bound behavior, which is outweighed by the pipelined speculation of LoopSpec, resulting in the end-to-end speedups reported in our experiments.

W3. Rationale for choosing the proposal depths in Table 1

We would like to clarify that Section 5 of the paper provides a theoretically motivated method for selecting the proposal depths (Thm 2). For the Raven model family with 32 loops, the theorem gives (d1⋆,d2⋆)=(1.26,8.85)(d_1^\star, d_2^\star)=(1.26,8.85). We empirically evaluate several nearby proposal depths configurations in Table 1. The results confirm that (d1,d2)=(2,8)(d_1, d_2)=(2,8) is the best configuration, which closely matches the theoretical result.

AuthorsOpenReview ↗

Reminder: Discussion Period Ending in a Few Days

Dear Reviewer SF7N,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In our response, we report speedups over 10 random seeds with standard deviations of only 0.02–0.07x, e.g., 2.40 ± 0.03x on GSM8K and 2.34 ± 0.06x on AIME25 for Ouro-1.4B-Thinking. We also profiled the branch cost on Raven-Llama: per-recurrent latency grows from 696 μs at 1 branch to 1,910 μs at 249 branches, i.e., 2.74x for a 249x increase in branches, with a mean branch size of 34.06. We clarified questions including the gating mechanism, the power-law envelope in Fig. 3(b), and the choice of proposal depths via Theorem 2. To address the model scope, we added a third architecture, Dory1.0-4B, with 2.29x speedup on GSM8K.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

I appreciate the authors for their further explanations and added experiments. My concerns have been addressed by the additional variance and cost experiments, and I am happy to raise my score accordingly.

Reviewer_asuwOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies looped transformers and proposed an inference-only mechanism to speed up their decoding and KV cache usage by using the earlier loop's generation as a speculative draft to be verified by the later loops (self-speculative decoding). The first proposal to do this: say there are R loops in the model. An intermediate depth d1 generates the draft tokens which are batched together with the processing of earlier tokens through deeper loops. However, this can end up wasting a lot of computation if the number of loops is large as we could draft ahead a lot of tokens based on an erroneous draft token. To ameliorate this, the authors use a cascaded form of speculative decoding where a deeper loop (d2) on token i produces a better quality draft for token i+1 (called 2nd proposal) and an alternate continuation is started (based on a gating mechanism) using this for future loops which has a higher chance of being accepted. Intuitively, the alternative (fallback) continuation is created only when the deeper loop d2 expresses low confidence in the first proposal token from d1. Finally when a token reaches depth R, verification is done in a cascaded manner: first the 1st proposal is verified, if accepted, we keep it and its descendants and prune away any 2nd proopsal if it exists. If this gets rejected and a 2nd proposal is available, we accept this and keep the 2nd proopsal's descendants discarding the first proposal branch. If both branches are rejected then we simply use the final token from R and continue decoding from scratch (as in normal speculative decoding).

The authors report up to a 6.8x lossless speedup across two looped model families: Raven (R = 32) and Ouro (R = 4) with LoopSpec.

Critical strengths
  1. The paper is well-written and experimental ablations are comprehensive. Especially the fact that speedups are real wall-clock times and not theoretical estimates.

  2. The savings are substantial and can represent significant speedups in decoding looped models in small batch size settings.

Critical weaknesses
  1. The idea of speculative decoding is not very novel and it seems pretty natural to try it in the space of looped transformers. That being said, actually implementing it and executing careful experiments is valuable to the community.

1.How do the speedups change across batch sizes will be an interesting datapoint to see.

Questions for authors
  1. How do the results change when our main batch size is not 1 but higher? This profiling graph across batch sizes will be interesting to see.
  2. will be nice to add comaprison to loopahead, DLoop paper. came out pretty recently so not expected but will be nice to do it now.
3 comments
AuthorsOpenReview ↗

Initial Response

We are deeply grateful to the reviewer for the positive evaluation and for recognizing the strengths of our work. We address the questions below.

Q1. LoopSpec in Larger Batch Size

We added multi-batch evaluations in Section 6.4. We measure decoding throughput under concurrent requests on Ouro-1.4B (d1=1,d2=2d_1=1, d_2=2) and Raven-Llama (d1=2,d2=8d_1=2, d_2=8) using GSM8K.

As shown in the tables, LoopSpec consistently achieves speedups across all evaluated batch sizes and sequence lengths. Specifically, at B=16, LoopSpec achieves 1.29–1.60x speedup on Ouro-1.4B and 1.84–2.68x speedup on Raven-Llama across the evaluated settings.

Ouro-1.4B — GSM8K

Sequence LengthB=1B=2B=4B=8B=16
5122.94x2.74x2.35x1.88x1.60x
10242.84x2.53x2.08x1.61x1.41x
20482.71x2.26x1.75x1.40x1.29x

Raven-Llama — GSM8K

Sequence LengthB=1B=2B=4B=8B=16
5125.61x5.39x4.71x3.23x2.68x
10245.45x5.08x4.24x2.81x2.29x
20485.22x4.60x3.58x2.30x1.84x

These results demonstrate that LoopSpec's acceleration extend beyond single-batch inference and remain effective under multi-batch scenarios.

Q2. Comparison vs LoopAhead and DLoop

LoopAhead builds on our pipelined framework and adds a trained lookahead drafter, whereas LoopSpec is training-free. DLoop targets standard speculative decoding with a separate draft model and is orthogonal to LoopSpec. Since neither implementation is publicly available yet, we will cite both works and add a comparison once their codes are released.

AuthorsOpenReview ↗

Reminder: Discussion Period Ending in a Few Days

Dear Reviewer asuw,

Just a brief reminder that the author–reviewer discussion period ends in a few days.

We updated the submission by adding the batch-size profiling you requested (Section 6.4). As a result, LoopSpec remains faster at every batch size from 1 to 16 and sequence length from 512 to 2048, with 1.29–1.60x speedup on Ouro-1.4B and 1.84–2.68x on Raven-Llama at B=16. We also clarified how LoopSpec relates to LoopAhead (which builds on our pipelined framework with a trained drafter) and DLoop (standard speculative decoding with a separate draft model). We will cite both and add a direct comparison once their code is released.

If you have any follow-up questions or remaining concerns, we would be happy to address them.

Thank you again for your positive evaluation and for the time and effort you put into the review.

Best regards,

Authors

Reviewer_asuwOpenReview ↗

Thanks for your response

Thank you to the authors for your response. I maintain my rating.

Reviewer_BJFwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LoopSpec, a training-free self-speculative decoding method for Looped Transformers, which reuse a shared block stack across many recurrent depths and are therefore slow to decode. It exploits the observation that intermediate-depth readouts already closely approximate the final distribution, treating an early readout at depth d1 as a draft token whose remaining depths serve as verification, while a gated second proposal at depth d2 drawn from the residual distribution provides a fallback. Cascade verification is designed to preserve the target distribution exactly, and two theorems derive the double-rejection bound and optimal depths in closed form.

Critical strengths
  1. A training-free, parameter-free method with large wall-clock gains in a real SGLang serving stack — up to 6.83× on Raven and 3.36× on Ouro (Table 1), beating a trained DFlash drafter by up to 1.7× (Fig. 5). First proposals supply over 91% of committed tokens and full-depth restarts account for under 0.5% (App. E).
  2. The residual second proposal makes the double-rejection bound independent of d1 (Theorem 1). I re-derived Lemma 4 and the gate-closed case (App. G.1) and reproduced the closed-form optimal depths for both Raven models (Sec. 5, App. F).
  3. Each component's delta is checkable: single-proposal → residual second proposal → gating adds incremental speedup (Table 3), and gating cuts the mean effective batch size from 43.75 to 34.06 (Fig. 4).
Critical weaknesses
  1. The novelty over prior pipelined speculation is not isolated. The core single-proposal pipeline (Sec. 4.2) is conceptually SPEED (Hooper et al., 2023) / PPD (Yang et al., 2024), and the paper's own numbers show modest incremental gains from the new components — roughly +6% to +22% on Raven, sometimes negative on Ouro (Table 1 greedy). The sampling ablation (Table 3) has no first-proposal-only row. External baselines are weak: LayerSkip runs in HF Transformers with an untuned draft length (App. D.2), and DFlash is trained for only 10k steps on R=4 models (App. D.1). Fix: first-proposal-only row under sampling; LayerSkip tuned in the same stack; precise statement of what is new beyond training-free and sampling support.
  2. "Lossless" is unverified and the evaluation is single-request only. No accuracy or greedy exact-match rate vs. AR is reported, and BF16 batch non-invariance across branch sizes (1 to 243, Fig. 4) means greedy outputs can drift. All results are single-batch on one GPU with models ≤2.6B (Sec. 6), with no multi-request serving, throughput or tail-latency measurement — branch FLOPs and per-branch KV caches may compete with other requests under load. Fix: AR vs. LoopSpec accuracy and greedy exact-match per benchmark; throughput, latency and peak memory at several concurrency levels.
  3. "Theory closely matches the measured optima" is only partly supported: Theorem 2 minimises an upper bound that ignores pre-/post-layer overhead (which the paper itself says drives the wall-clock gap on Raven, p. 8), only five configurations are tested, and d1=1 — the rounded prediction — is never evaluated. Fix: a (d1, d2) grid including d1=1, predicted vs. measured.
Questions for authors
  1. If AR vs. LoopSpec accuracy is unchanged within noise and greedy outputs match AR on nearly all sequences, I would raise my score; if they diverge materially, the "lossless" claim is unsupported and I would lower it.
  2. If LoopSpec keeps a clear margin over a first-proposal-only baseline and over LayerSkip tuned in the same stack under both greedy and sampling, I would raise my score; if the gap mostly closes, the gain comes from the implementation rather than the method and I would lower it.
3 comments
AuthorsOpenReview ↗

Initial Response (1/2)

We sincerely thank the reviewer for the thorough review and critical insights. We take this opportunity to clarify some misunderstandings and provide additional experiments as below.

Q1/W2-1. "Lossless" of LoopSpec

We firstly would like to clarify that speculative decoding is lossless by construction [1][2], and LoopSpec follows such construction which guarantees "lossless" mathematically. However, it is known that serving kernels are not batch-invariant [3]. LoopSpec batches tokens at different recurrent depths, so it runs each token in different batch shapes than AR decoding. The reduction order in matmul, RMSNorm, and attention depends on batch size and composition, so the same token can receive slightly different logits between LoopSpec and AR decoding. Such numerical differences are a hardware-level effect, and overall it does not change the underlying speculative decoding algorithm to be lossless.

Furthermore, to rule out these numerical effects, we ran Ouro-1.4B and Raven-Llama with SGLang's --enable-deterministic-inference mode. Following [3], this mode uses batch-invariant kernels with fixed reduction orders and fixed KV split sizes, so each token's result does not depend on the batch it runs in. In this setting, LoopSpec produces token-identical outputs to AR on every problem under greedy scenario and preserves full accuracy of AR:

ModelTaskAR accuracyLoopSpec accuracyIdentical GenerationsTotal # output tokens
Ouro-1.4BGSM8K78.77%78.77%1319/1319142,450
Ouro-1.4BMBPP+65.08%65.08%378/37821,102
Raven-LlamaGSM8K52.46%52.46%1319/1319118,368
Raven-LlamaMBPP+24.87%24.87%378/37854,403

In total, 3394/3394 generations match exactly, and achieve the same accuracy across benchmarks, showing the losslessness of LoopSpec.

[1] Y. Leviathan, M. Kalman, Y. Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. [2] C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, J. Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318, 2023. [3] H. He and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, Sep 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Q2-1/W1-1. First Proposal Only Ablation

We extend Table 3 with a first proposal only setting to isolate the contributions of LoopSpec's additional components:

Raven-LlamaGSM8KMATH500HumanEval+MBPP+
+ First proposal only2.66x3.75x3.87x3.44x
+ First & Non-residual Second Proposal3.24x4.49x4.59x4.07x
+ Residual Second Proposal3.52x4.73x4.76x4.57x
+ Gating Mechanism3.74x4.85x5.00x4.71x

Overall, LoopSpec with residual second proposal and the gating mechanism achieves 1.29-1.41x speedup over first proposal only baseline. These results show that the additional components provide significant performance gains beyond the first proposal only baseline. We updated Table 3 to include this result and highlight the contributions of each component.

AuthorsOpenReview ↗

Initial Response (2/2)

Q2-2/W1-2. Same Stack Comparison of LayerSkip with Tuned Draft Length

We clarify that we conducted the comparison between LoopSpec and LayerSkip under a fair setting, both under the same stack of HuggingFace Transformers.

To address the reviewer's concern about LayerSkip's draft length, we additionally evaluate 3 typical draft lengths (8, 12, and 16) of LayerSkip, and compare it against LoopSpec under the same stack for both greedy and sampling setting:

ModelGreedy/SamplingDraft LengthGSM8KMBPP+
Ouro-1.4BGreedy81.82x1.95x
121.69x1.83x
161.51x1.67x
LoopSpec2.28x2.43x
Sampling (T=1, top-p=0.7)81.50x1.69x
121.31x1.53x
161.14x1.39x
LoopSpec2.18x2.35x
Raven-LlamaGreedy82.93x3.21x
123.03x3.42x
162.99x3.66x
LoopSpec4.25x4.69x
Sampling (T=1, top-p=0.7)82.77x3.16x
122.81x3.41x
162.68x3.54x
LoopSpec3.38x4.25x

As shown in the table, LoopSpec consistently outperforms LayerSkip across models, benchmarks, and tuned draft length by 1.20-1.45x. These results demonstrate LoopSpec's performance advantage over LayerSkip under the same stack and tuned LayerSkip's draft length, showing LoopSpec's gains come from the method itself rather than from implementation details.

W1-3. DFlash Comparison

We follow the training recipe provided by the official SGLang SpecForge [4]. To further rule out the bias for our training recipe, a concurrent work LoopAhead [5] also trains 10k steps for DFlash, identical to our setting, and reports that training converges before 10k steps. Also, we emphasize that LoopSpec is a plug-and-play training free method, while DFlash requires a large amount of resources to train and an extra drafter, therefore LoopSpec is better than DFlash in terms of both efficiency and speedup.

[4] SGLang Team. SpecForge: A Framework for Speculative Decoding Training. GitHub, 2026. https://github.com/sgl-project/SpecForge#-supported-methods [5] Z. Liu, N. Xue, R. Bao, Y. Xu, X. Cao, J. Yao, Z. Chen. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models. arXiv:2610.21594, 2026.

W2-2. Multi-Request Serving

We added multi-batch evaluations in Section 6.4. We measure decoding throughput under concurrent requests on Ouro-1.4B (d1=1,d2=2d_1=1, d_2=2) and Raven-Llama (d1=2,d2=8d_1=2, d_2=8) using GSM8K.

As shown in the tables, LoopSpec consistently achieves speedups across all evaluated batch sizes and sequence lengths. Specifically, at B=16, LoopSpec achieves 1.29–1.60x speedup on Ouro-1.4B and 1.84–2.68x speedup on Raven-Llama across the evaluated settings.

Ouro-1.4B — GSM8K

Sequence LengthB=1B=2B=4B=8B=16
5122.94x2.74x2.35x1.88x1.60x
10242.84x2.53x2.08x1.61x1.41x
20482.71x2.26x1.75x1.40x1.29x

Raven-Llama — GSM8K

Sequence LengthB=1B=2B=4B=8B=16
5125.61x5.39x4.71x3.23x2.68x
10245.45x5.08x4.24x2.81x2.29x
20485.22x4.60x3.58x2.30x1.84x

These results demonstrate that LoopSpec's acceleration extend beyond single-batch inference and remain effective under multi-batch scenarios.

We hope these results could address your concerns. If not, further clarification will help a lot. Once again, we deeply appreciate your review and comments, and look forward to your feedback on our response.

AuthorsOpenReview ↗

Reminder: Discussion Period Ending in a Few Days

Dear Reviewer BJFw,

Just a brief reminder that the author–reviewer discussion period concludes in a few days.

In your review, you said you would raise your score if LoopSpec preserved AR accuracy and greedy outputs, and if it kept a clear margin over a first-proposal-only baseline and over LayerSkip in the same stack. Both conditions hold in our response. With SGLang's deterministic inference mode, LoopSpec produces token-identical outputs to AR on 3394/3394 generations (GSM8K and MBPP+, Ouro-1.4B and Raven-Llama) with identical accuracy, e.g., 78.77% vs 78.77% on Ouro GSM8K and 24.87% vs 24.87% on Raven MBPP+. We added the first-proposal-only row to Table 3, where the full LoopSpec is 1.29–1.41x faster on Raven-Llama. We also tuned LayerSkip's draft length (8/12/16) in the same HuggingFace stack, and LoopSpec still outperforms it by 1.20–1.45x under both greedy and sampling. In addition, we added multi-request throughput results for batch sizes 1–16 and sequence lengths 512–2048 (Section 6.4), where LoopSpec reaches 1.84–2.68x on Raven-Llama at B=16.

If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider raising your rating as you indicated. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.

Thank you again for the time and effort you put into the review.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.