acceptodds
Under review as a conference paper at ICLR 2027

ProSpect: Efficient and Consistent Self-Speculative Decoding

Abstract

In single-user and on-device settings, large language model (LLM) decoding typically runs at batch size one and is memory-bandwidth bound. Every forward pass streams the full model weights to emit a single token, leaving the GPU's arithmetic idle. Self-speculative decoding turns that idle arithmetic into tokens without adding any weights, by drafting the next block of tokens in parallel and verifying the previous block autoregressively in a single pass. There is no auxiliary drafter to train, keep aligned with the target, or hold in memory beside it, and no ceiling on draft quality set by a smaller model's capacity. We address two shortcomings of self-speculation. First, parallel drafts are sampled independently and can be individually plausible yet jointly incoherent. By front-loading the randomness of sampling, we generate consistent drafts without sacrificing parallelism. Second, a decode step cannot know how many of its drafts will be accepted, so a draft branch has to be appended for every possible acceptance count, which makes the query size quadratic in the draft length. By drafting only for the most likely acceptance counts, we reduce the query size to linear. The resulting decoder, ProSpect decodes faster than autoregressive Qwen3-8B on SpeedBench, outperforming speculative decoding with dedicated drafters DSpark and DFlash by % and %, and faster than autoregressive Gemma-4-31B, ahead of DFlash by %.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, self-speculation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gGLWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProSpect, a self-speculative decoding method in which a single model drafts and verifies in one forward pass. It makes two contributions. First, Gumbel-noise conditioning represents pre-sampled noise as mask tokens and reuses it at verification, so parallel drafts approximate the joint conditional rather than per-position marginals. Second, a dynamic draft tree verifies multiple candidates per position and places a fixed number of branches at the most likely landing points, keeping the query size linear rather than quadratic in draft length. Implemented in vLLM with a FlashAttention cascade, ProSpect reports 3.9× speedup on Qwen3-8B and 3.1× on Gemma-4-31B on SpeedBench at batch size 1, outperforming DFlash, DSpark, and Gemma-4 MTP.

Critical strengths
  1. Simple and effective idea for consistent parallel drafting. Encoding pre-sampled Gumbel noise as ordinary tokens and reusing it at verification lets a single forward pass draft from approximate conditionals rather than marginals. It also makes verifying multiple candidates at the same position trivial. The ablation (Table 3) shows clear acceptance gains that grow with draft length.

  2. Practical, self-contained system. Combining a linear-size dynamic draft tree with a FlashAttention cascade in vLLM avoids both the quadratic query of prior self-speculation and the cost of serving a separate drafter. Training only touches LoRA adapters and mask-token embeddings, and the method delivers strong batch-size-1 speedups on two model families.

Critical weaknesses
  1. Lossy speedups compared against lossless baselines. The headline speedups come from ProSpect†, whose merged adapter changes the verifier and costs up to ~4 points on Qwen3-8B. The exact (gated) variant is slower than DSpark, so the claimed advantage over DFlash and DSpark depends on giving up exactness.
  2. Quantized sampler. The fixed set of 10 noise vectors replaces temperature sampling with a quantized sampler that can emit at most 10 tokens per position. This target is easier to draft for than the one the baselines accelerate, and its bias is never quantified.
  3. Repetition loops in long outputs. With thinking enabled, ProSpect falls into repetition loops several times more often than lossless methods (up to 16% of LiveCodeBench turns), which makes the accelerated model substantially less usable in exactly the long-reasoning settings it targets.
  4. Favorable experimental setup. Tree configurations are tuned on the evaluation sets, with a different one per benchmark suite, while baselines use default settings. The Qwen3-8B DFlash and DSpark drafters were not trained on thinking data, which inflates the thinking-mode margins.
Questions for authors

Please see the weaknesses.

2 comments

We thank the reviewer for the review.

W1 (lossy speedups). Table 1 reports both variants: gated ProSpect is bit-identical to the base model, and the merged variant is marked as not exact. The merged variant reduces average accuracy by 1.26 points on Qwen3-8B and 0.32 points on Gemma-4-31B. Its largest single-benchmark difference is on GPQA-Main for Qwen3-8B, which corresponds to the ~4 points mentioned by the reviewer. The accuracy gap is caused by fine-tuning, not by speculation. Verification in ProSpect is exact with respect to the weights it runs, so every emitted token is the token the fine-tuned model itself would produce. We fine-tune on the models' own generations because the post-training data of Qwen3 and Gemma-4 are not public. If ProSpect were trained jointly with the model's post-training, the released model would itself be the verifier, and decoding with ProSpect would be bit-identical to decoding the released model. With access to the post-training data, we expect the accuracy gap to disappear.

Among methods that do not preserve the base model's outputs, ProSpect is both more accurate and faster. The I-DLM conversion of the same Qwen3-8B weights is less accurate (Table 2: 81.46 vs. 81.88) and about half as fast (Table 1: 1.88× vs. 3.88×). Nemotron-Labs-Diffusion-8B, a natively trained diffusion model, scores lower than ProSpect on every benchmark it reports and is slower in both of its decoding modes.

W2 (quantized sampler). Given the context, a position can indeed emit at most LL distinct tokens. We quantified the resulting bias. We evaluated the ProSpect checkpoints on the reasoning benchmarks of Table 2 with noise sets of size L∈{1,5,10,100}L \in \{1, 5, 10, 100\} and with unrestricted Gumbel noise, over the same five seeds. Eight-benchmark average accuracy (%):

Qwen3-8BGemma-4-31B
Unrestricted noise81.39 ± 0.5388.08 ± 0.32
L=1L=177.47*87.26*
L=5L=581.45 ± 0.4887.54 ± 0.31
L=10L=10 (deployed)81.34 ± 0.5088.10 ± 0.15
L=100L=10081.37 ± 0.3188.00 ± 0.37

*With one noise vector the seed has no effect, so the five runs are a single deterministic decode.

Every L>1L>1 is within noise of unrestricted sampling, except for a small drop at L=5L=5 on Gemma-4-31B. Only L=1L=1 reduces accuracy clearly. The gap to the base model in Table 2 is already present with unrestricted noise, so it is caused by the adapter and not by the finite noise set.

W3 (repetition loops). Generations with repetition loops are removed from all speed measurements, for all methods, so they do not affect the reported throughput or acceptance. The number of removed turns per category and method is given in the Appendix (Tables 10, 14, 18 and 22). The loops occur with the Qwen3-8B adapter. Gemma-4-31B ProSpect shows no increase in repetition: across both suites and both thinking modes, about as few of its turns are removed as for its base model. The accuracies in Table 2 are computed over all generations without filtering, so the effect of the loops is included there.

W4 (experimental setup). Each model uses one tree configuration for all categories of a benchmark suite, selected from the sweep in the Appendix. The conclusion does not depend on this selection. All twelve dynamic tree configurations in the Qwen3-8B sweep (Appendix, Table 6) are faster than DFlash and DSpark in both thinking modes, as are all static configurations except K=7K=7 with thinking disabled, which is on par with DSpark. As noted in Table 1, the released Qwen3-8B drafters were not trained on thinking data. With thinking disabled, where this does not apply, ProSpect reaches 3.88×, compared with 3.02× for DSpark.

If our responses address your concerns, we would appreciate it if you would consider raising your score. Please let us know if any concerns remain before the end of the discussion period.

AuthorsOpenReview ↗

Improvements to Gated ProSpect

Since submission, we have improved the gated variant in two ways. Its verifier remains bit-identical to the base model. First, the gated adapter now computes its draft-row gate on the GPU, which removes a host synchronization at every step, and fuses the LoRA expansion with the gated addition into a single kernel. At the tree configuration of the paper, this lowers the step time from 11.4 to 9.7ms with thinking disabled, without changing the acceptance. Second, we continued training the gated adapter of the paper on its data together with non-thinking responses to the same prompts, which raises the accepted tokens per step by up to 3.5%. Under the protocol of Table~1 (Qwen3-8B, SpeedBench, H100, same tree configuration), gated ProSpect now outperforms DFlash and DSpark in both thinking modes:

Thinking disabled

tok/stepms/steptok/secSpeedup
Autoregressive1.006.8146.41.00×
DFlash3.598.9397.02.71×
DSpark3.799.1442.13.02×
Gated ProSpect, submission3.9611.4363.52.48×
Gated ProSpect, updated4.109.7455.23.11×

Thinking enabled

tok/stepms/steptok/secSpeedup
Autoregressive1.007.0143.51.00×
DFlash2.749.1302.02.10×
DSpark3.309.0392.32.73×
Gated ProSpect, submission3.9811.1402.82.81×
Gated ProSpect, updated4.0110.0459.63.20×

On Qwen3-8B, distribution preservation therefore no longer requires a trade-off: the lossless gated variant is faster than every distribution-preserving baseline in both thinking modes.

Reviewer_iUAROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper targets two important limitations of parallel speculative decoding: dependencies among draft tokens and efficient allocation of a fixed drafting budget.
  2. Gumbel-noise conditioning is an interesting mechanism for capturing cross-token dependencies.
  3. The self-speculative design avoids training and serving a separate drafter, making the approach comparatively simple to deploy.
Critical strengths
  1. The paper targets two important limitations of parallel speculative decoding: dependencies among draft tokens and efficient allocation of a fixed drafting budget.
  2. Gumbel-noise conditioning is an interesting mechanism for capturing cross-token dependencies.
  3. The self-speculative design avoids training and serving a separate drafter, making the approach comparatively simple to deploy.
Critical weaknesses
  1. While the idea of "Gumbel-noise conditioning" is interesting, the core concept is largely adopted from the cited work of Zhang et al. (2026). The main differences are ProSpect's runtime-efficient, finite-set-based method for simplifying Gumbel-noise injection and its reuse of the same noise for runtime efficiency; however, the finite-set approximation raises another concern (described in point #2).

  2. The proposed implementation of the Gumbel-Max scheme uses only 10 predefined Gumbel-noise vectors, which discretizes the sampling randomness and may distort the sampling distribution. Although the authors state that this introduces only slight bias, this claim requires much more thorough validation than a single sentence.

  3. For example, Table 2 shows a mostly consistent drop from Base to ProSpect, which could result from the combined effects of the finite-set Gumbel scheme and adapter training. Thus, the table does not isolate the potential threat / negative impact of the finite-set approximation.

  4. (related to #3) Furthermore, the evaluated tasks are mostly reasoning tasks, where a model may still derive the correct final answer despite perturbations to the sampling distribution. Although its mechanism differs a bit, recent work [1] reports that random embedding injection can affect, and in some settings improve, reasoning accuracy. A stronger evaluation of the "slight bias" claim is encouraged to include direct distributional measurements, such as generation-length distributions (e.g. there might be a chance that noise -> longer generation -> preserving final accuracy), etc.

  5. Distribution preservation is a critical aspect of speculative decoding in practical applications; practitioners may not want to risk altering the target distribution. The main ProSpect configuration alters the target model's behavior because its adapter is active during verification, which may limit its practical value. I am not sure whether the throughput gain over distribution-preserving baselines (e.g. 109.3 to 122.2 tok/s on Gemma-4-31B even excluding the Gemma's own MTP model for a fairer comparison) is sufficient to justify this trade-off.

  6. Table 1 appears incomplete. For instance, Gated ProSpect is not reported for Gemma-4-31B. In addition, the Nemotron block does not include ProSpect on the same backbone so it does not provide a direct comparison with the proposed method.

[1] "From Noise to Diversity: Random Embedding Injection in LLM Reasoning," arXiv:2605.11936.

Questions for authors
  1. Out of curiosity, in Figure 4 (right), if verification lands on York, could it sometimes be beneficial to truncate the accepted prefix at New and use New's draft branch, rather than accept York and incur a cold step that requires rebuilding the draft tree? Is this possible?

  2. Have you done any experiments using batch size > 1?

2 comments

We thank the reviewer for the review.

W1 (relation to Zhang et al.). The conditioning builds on Zhang et al., as stated in the paper. Our modifications are the finite set of noise tokens, which replaces the learned noise encoder with embedding rows, and the reuse of the noise at verification, which reduces acceptance to an equality check. The dynamic draft tree is a separate contribution. In Table 3 it accounts for a throughput gain comparable to that of the conditioning.

W2 and W3 (finite noise set, and separating it from the adapter). We evaluated the ProSpect checkpoints on the reasoning benchmarks of Table 2 with noise sets of size L∈{1,5,10,100}L \in \{1, 5, 10, 100\} and with unrestricted Gumbel noise, over the same five seeds. Eight-benchmark average accuracy (%):

Qwen3-8BGemma-4-31B
Unrestricted noise81.39 ± 0.5388.08 ± 0.32
L=1L=177.47*87.26*
L=5L=581.45 ± 0.4887.54 ± 0.31
L=10L=10 (deployed)81.34 ± 0.5088.10 ± 0.15
L=100L=10081.37 ± 0.3188.00 ± 0.37

*With one noise vector the seed has no effect, so the five runs are a single deterministic decode.

Every L>1L>1 is within noise of unrestricted sampling, except for a small drop at L=5L=5 on Gemma-4-31B. Only L=1L=1 reduces accuracy clearly. The gap to the base model in Table 2 is already present with unrestricted noise, so it is caused by the adapter and not by the finite noise set.

W4 (generation length). The Appendix reports the mean generation length per category for every method, in both thinking modes and on both benchmark suites (Tables 9, 13, 17 and 21). After filtering repetitive turns, the generation lengths of ProSpect are close to those of the base model in most categories. The accuracy is therefore not maintained through longer generations.

W5 (distribution preservation). Table 1 reports both variants: gated ProSpect is bit-identical to the base model, and the merged variant is marked as not exact. The merged variant reduces average accuracy by 1.26 points on Qwen3-8B and 0.32 points on Gemma-4-31B. This gap is caused by fine-tuning, not by speculation. Verification in ProSpect is exact with respect to the weights it runs, so every emitted token is the token the fine-tuned model itself would produce. We fine-tune on the models' own generations because the post-training data of Qwen3 and Gemma-4 are not public. If ProSpect were trained jointly with the model's post-training, the released model would itself be the verifier, and decoding with ProSpect would be bit-identical to decoding the released model. With access to the post-training data, we expect the accuracy gap to disappear. On Gemma-4-31B, where the throughput margin over distribution-preserving baselines is smaller, the accuracy gap is also smaller.

Among methods that do not preserve the base model's outputs, ProSpect is both more accurate and faster. The I-DLM conversion of the same Qwen3-8B weights is less accurate (Table 2: 81.46 vs. 81.88) and about half as fast (Table 1: 1.88× vs. 3.88×). Nemotron-Labs-Diffusion-8B, a natively trained diffusion model, scores lower than ProSpect on every benchmark it reports and is slower in both of its decoding modes.

W6 (Nemotron block). The Nemotron-Labs-Diffusion rows serve as a reference for a natively trained diffusion model of the same size, run in its two released decoding modes. They are not a comparison of ProSpect on that backbone.

Q1 (truncating at New). We tested this policy, and it does not improve throughput. Under the same noise, the drafts in the branch of New largely reproduce the continuation that was rejected, so the next step often fails at the same position.

Q2 (batch size > 1). ProSpect is designed for the memory-bound regime, where the forward pass has unused compute that can be spent on draft tokens. Each request adds its own query block, so at large batch sizes the forward pass becomes compute bound and this spare capacity is no longer available. This applies to speculative decoding in general and is stated in the Limitations.

If our responses address your concerns, we would appreciate it if you would consider raising your score. Please let us know if any concerns remain before the end of the discussion period.

AuthorsOpenReview ↗

Improvements to Gated ProSpect

Since submission, we have improved the gated variant in two ways. Its verifier remains bit-identical to the base model. First, the gated adapter now computes its draft-row gate on the GPU, which removes a host synchronization at every step, and fuses the LoRA expansion with the gated addition into a single kernel. At the tree configuration of the paper, this lowers the step time from 11.4 to 9.7ms with thinking disabled, without changing the acceptance. Second, we continued training the gated adapter of the paper on its data together with non-thinking responses to the same prompts, which raises the accepted tokens per step by up to 3.5%. Under the protocol of Table~1 (Qwen3-8B, SpeedBench, H100, same tree configuration), gated ProSpect now outperforms DFlash and DSpark in both thinking modes:

Thinking disabled

tok/stepms/steptok/secSpeedup
Autoregressive1.006.8146.41.00×
DFlash3.598.9397.02.71×
DSpark3.799.1442.13.02×
Gated ProSpect, submission3.9611.4363.52.48×
Gated ProSpect, updated4.109.7455.23.11×

Thinking enabled

tok/stepms/steptok/secSpeedup
Autoregressive1.007.0143.51.00×
DFlash2.749.1302.02.10×
DSpark3.309.0392.32.73×
Gated ProSpect, submission3.9811.1402.82.81×
Gated ProSpect, updated4.0110.0459.63.20×

On Qwen3-8B, distribution preservation therefore no longer requires a trade-off: the lossless gated variant is faster than every distribution-preserving baseline in both thinking modes.

Reviewer_ndaxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProSpect, a self-speculative decoding method that conditions parallel draft generation on Gumbel noise, using gated LoRA fine-tuning to improve draft generation. The approach aims to mitigate the inefficiency of parallel drafting caused by uncertainty in the number of tokens accepted during verification.

Critical strengths

The method does not require a separate draft model, which is attractive for memory-constrained deployment.

The paper explores both merged and gated LoRA implementations, highlighting a practical trade-off between inference efficiency and output fidelity.

Critical weaknesses
  1. Limited novelty relative to prior work. The core idea appears closely related to Parallel Token Prediction for Language Models (ICLR 2026), which reportedly explores random variables as drafter input and gated LoRA for fine-tuning. The paper needs to clearly distinguish its technical contributions from this prior work. Without a precise comparison, the novelty of the proposed method is difficult to establish.

  2. The finite-noise approximation is insufficiently validated. ProSpect replaces the idealized Gumbel-noise distribution with only 10 fixed noise vectors. The authors acknowledge that this introduces a slight bias, but claim that it does not affect generation quality without providing quantitative evidence. This leaves an important question unanswered: how does the finite noise set affect the draft distribution, generation quality, and decoding performance? The paper should quantify this effect and evaluate sensitivity to the number of noise vectors.

  3. The speedup comparison does not consistently reflect output fidelity. The merged implementation modifies the model weights and is therefore not bit-identical to the original model, whereas the gated implementation preserves the original model weights but incurs additional computation (11.4 vs. 8.2 ms/step). Comparing the potentially lossy merged implementation against lossless baselines can overstate the practical advantage of ProSpect. Under the lossless setting, the reported 2.48× speedup of gated ProSpect is below DFlash (2.71×) and DSpark (3.02×). The paper should make the fidelity–latency trade-off explicit and compare methods under consistent output-quality and verification settings.

  4. The evaluation methodology raises concerns about the reliability of the reported performance claims. The paper appears to aggregate throughput (tokens/s) using token-weighted pooled values, while reporting tokens/step and latency/step as averages across categories. This inconsistency can bias aggregate comparisons, particularly in favor of ProSpect. Moreover, the speed measurements lack estimates of variability, making it difficult to determine whether the reported differences are robust. The evaluation should use clearly defined, consistently aggregated metrics and report measurement variability.

Questions for authors
  1. Effect of the finite noise set. The method uses only 10 fixed noise vectors and acknowledges that this introduces a slight bias, yet reports no quantitative evidence supporting the claim that generation quality is unaffected. Could the authors provide quantitative results on output quality and decoding performance under different noise-set sizes, including a comparison with a less restrictive noise construction? This would help establish whether the finite-noise approximation materially affects the method's effectiveness.

  2. Comparison under consistent output-fidelity constraints. The merged implementation modifies the model weights, whereas the gated implementation preserves the original weights at additional computational cost. Could the authors clarify whether the reported speedups are measured under comparable output-fidelity and verification settings, and provide a consistent comparison with DFlash and DSpark using the lossless gated implementation? This clarification could materially affect the assessment of ProSpect's practical advantage.

  3. Whether the proposed conditioning addresses the stated motivation. The paper argues that independent parallel drafting produces implausible token combinations and that uncertainty in the number of accepted tokens leads to unnecessarily long drafts. Could the authors provide evidence that Gumbel-noise conditioning directly improves these aspects, for example through draft-token plausibility, acceptance behavior, or effective draft length, compared with the unconditioned baseline? This would clarify whether the proposed mechanism addresses the motivating problem rather than merely changing the draft-generation process.

3 comments

We thank the reviewer for the review.

W1 (relation to PTP). We will cite PTP. PTP feeds a uniform auxiliary variable per position to a fine-tuned drafter through a binary embedding, selects the token by the inverse CDF, reuses the variable at verification, and uses one draft branch per acceptance count. ProSpect differs in three respects. (i) The noise is a Gumbel vector from a fixed set of LL vocabulary tokens, so no additional embedding scheme is needed. (ii) The dynamic draft tree verifies several candidates per position and places a fixed number of branches at the most likely landing points, so the query grows linearly rather than quadratically with the draft length. In Table 3, replacing the dynamic tree with the best static tree reduces throughput by 22% and 21%, and removing the conditioning by 17% and 21%. (iii) Drafting and verification run in a single forward pass in vLLM with a FlashAttention cascade, evaluated on 8B and 31B models. We do not claim gated LoRA as a contribution.

W2 and Q1 (finite noise set). We evaluated the ProSpect checkpoints on the reasoning benchmarks of Table 2 with noise sets of size L∈{1,5,10,100}L \in \{1, 5, 10, 100\} and with unrestricted Gumbel noise, over the same five seeds. Eight-benchmark average accuracy (%):

Qwen3-8BGemma-4-31B
Unrestricted noise81.39 ± 0.5388.08 ± 0.32
L=1L=177.47*87.26*
L=5L=581.45 ± 0.4887.54 ± 0.31
L=10L=10 (deployed)81.34 ± 0.5088.10 ± 0.15
L=100L=10081.37 ± 0.3188.00 ± 0.37

*With one noise vector the seed has no effect, so the five runs are a single deterministic decode.

Every L>1L>1 is within noise of unrestricted sampling, except for a small drop at L=5L=5 on Gemma-4-31B. Only L=1L=1 reduces accuracy clearly. The gap to the base model in Table 2 is already present with unrestricted noise, so it is caused by the adapter and not by the finite noise set.

W3 and Q2 (fidelity). Table 1 reports both variants: gated ProSpect is bit-identical to the base model, and the merged variant is marked as not exact. The merged variant reduces average accuracy by 1.26 points on Qwen3-8B and 0.32 points on Gemma-4-31B. This gap is caused by fine-tuning, not by speculation. Verification in ProSpect is exact with respect to the weights it runs, so every emitted token is the token the fine-tuned model itself would produce. We fine-tune on the models' own generations because the post-training data of Qwen3 and Gemma-4 are not public. If ProSpect were trained jointly with the model's post-training, the released model would itself be the verifier, and decoding with ProSpect would be bit-identical to decoding the released model. With access to the post-training data, we expect the accuracy gap to disappear.

Among methods that do not preserve the base model's outputs, ProSpect is both more accurate and faster. The I-DLM conversion of the same Qwen3-8B weights is less accurate (Table 2: 81.46 vs. 81.88) and about half as fast (Table 1: 1.88× vs. 3.88×). Nemotron-Labs-Diffusion-8B, a natively trained diffusion model, scores lower than ProSpect on every benchmark it reports and is slower in both of its decoding modes.

W4 (aggregation). All three metrics in Table 1 are computed over the same set of filtered turns: throughput is total tokens divided by total wall time, tokens/step is the mean acceptance length per request, and ms/step is the mean decode-step time. The caption incorrectly describes these as category averages, and we will correct it. To test whether pooling favors ProSpect, we also computed each method's throughput as the unweighted mean of its eleven filtered per-category throughputs (Appendix, Tables 8 and 12). ProSpect is the fastest method in all four settings under this aggregation as well:

Speedup, per-category meanDFlashDSpark / Gemma-4 MTPProSpect
Qwen3-8B, thinking disabled2.70×2.98×3.77×
Qwen3-8B, thinking enabled2.09×2.58×3.81×
Gemma-4-31B, thinking disabled2.83×3.11×3.13×
Gemma-4-31B, thinking enabled3.47×3.49×3.71×

Q3 (effect of the conditioning). Table 3 isolates it: the two adapters share data, training recipe and query layout and differ only in the conditioning. The conditioning increases the accepted tokens per step in every layout, from 3.49 to 4.13 in the full configuration, and the increase grows with the draft length. Figure 3 (left) shows that the conditioned drafter has a lower branching factor than the marginal-only drafter and DFlash at every draft offset.

If our responses address your concerns, we would appreciate it if you would consider raising your score. Please let us know if any concerns remain before the end of the discussion period.

AuthorsOpenReview ↗

Improvements to Gated ProSpect

Since submission, we have improved the gated variant in two ways. Its verifier remains bit-identical to the base model. First, the gated adapter now computes its draft-row gate on the GPU, which removes a host synchronization at every step, and fuses the LoRA expansion with the gated addition into a single kernel. At the tree configuration of the paper, this lowers the step time from 11.4 to 9.7ms with thinking disabled, without changing the acceptance. Second, we continued training the gated adapter of the paper on its data together with non-thinking responses to the same prompts, which raises the accepted tokens per step by up to 3.5%. Under the protocol of Table~1 (Qwen3-8B, SpeedBench, H100, same tree configuration), gated ProSpect now outperforms DFlash and DSpark in both thinking modes:

Thinking disabled

tok/stepms/steptok/secSpeedup
Autoregressive1.006.8146.41.00×
DFlash3.598.9397.02.71×
DSpark3.799.1442.13.02×
Gated ProSpect, submission3.9611.4363.52.48×
Gated ProSpect, updated4.109.7455.23.11×

Thinking enabled

tok/stepms/steptok/secSpeedup
Autoregressive1.007.0143.51.00×
DFlash2.749.1302.02.10×
DSpark3.309.0392.32.73×
Gated ProSpect, submission3.9811.1402.82.81×
Gated ProSpect, updated4.0110.0459.63.20×

On Qwen3-8B, distribution preservation therefore no longer requires a trade-off: the lossless gated variant is faster than every distribution-preserving baseline in both thinking modes.

I appreciate the authors' serious efforts to address my concerns. The additional experiments have alleviated concerns regarding the finite noise set, throughput aggregation, and gated implementation. However, some key experiments and implementation improvements were only introduced during the rebuttal phase. The experimental completeness of the original submission, its novelty positioning, and the gaps between its claims and supporting evidence still affect my overall assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.