acceptodds
Under review as a conference paper at ICLR 2027

Self-Speculation for Faster Reasoning Models

Abstract

Reasoning language models generate long chains of thought (CoT) before responding, and for long-form outputs such as code, generating the final response accounts for a substantial share of end-to-end latency. Speculative decoding can accelerate this phase, but existing drafters either copy text the model has already written, as in prompt lookup, or use a cheaper approximation of the model, like a small draft model, to guess a few tokens ahead. Reasoning models expose an additional source of speculation: complete responses produced by the model from a partial CoT, which increasingly overlap with the final response as reasoning progresses. We introduce SSR, a training-free self-speculative decoding method that uses a model's own partial-CoT response to accelerate generation of its final response, deriving both draft and verifier from the same model at different reasoning budgets. The draft is generated concurrently with the remaining reasoning, hiding most of its cost, and is reused through prefix verification and suffix decoding to speculate the final response. SSR accelerates the response rather than reasoning and requires no auxiliary models. Across three model families (Gemma 4, Qwen3, and GPT-OSS), SSR achieves 1.23–1.55× speedups over standard decoding on long-form code generation (ClassEval) with comparable task accuracy; the draft alone yields 1.22–1.27×, and on top of suffix decoding with the same lookup mechanism, SSR adds a further 6–8% on Gemma and Qwen. We also analyze when SSR helps, including the effect of draft timing and length and where the reused text comes from. Code will be released upon publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, self-speculative decoding, inference acceleration, reasoning language models, chain-of-thought, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_55cKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SSR, a training-free method that drafts a response from a partial chain of thought while the same model continues reasoning, then reuses the draft through prefix verification and suffix decoding to accelerate the final response. Across three model families on ClassEval, SSR reports 1.23–1.55× end-to-end speedups over standard decoding. Compared with suffix decoding, the additional draft provides approximately 8.3% and 6.1% gains on Gemma and Qwen, respectively, while GPT-OSS shows no statistically clear incremental gain.

Critical strengths

1.Drafting from partial reasoning introduces candidate content not yet present in the context, extending the sources available to text-reuse methods.

2.SSR reuses model weights and prefix caches without training a separate drafter, while suffix reuse recovers useful content beyond the first prefix mismatch.

Critical weaknesses

1.SSR mainly introduces a new draft source, with limited methodological advances; targeted system-level optimizations could help deliver more substantial acceleration.

2.SSR only accelerates final responses, yet provides approximately 6–8% additional speedup over already model-free suffix decoding on Gemma and Qwen, despite requiring extra full-model drafting.

3.Omitted residual correction, filtered quality results, and a single ClassEval timing run without warmup leave quality preservation and performance robustness insufficiently established.

Questions for authors

1.What capabilities does partial-CoT drafting offer beyond SparseSpec and SuffixDecoding, and which workloads demonstrate substantial benefits?

2.What additional GPU time and memory does drafting require, and does its advantage over suffix decoding persist under concurrent workloads?

3.Do the incremental gains persist with residual-corrected verification, full-task quality evaluation, and repeated timing measurements?

Reviewer_UNJuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SSR, a training-free method for accelerating the final response generation of reasoning models. SSR generates a draft from partial CoT while the model continues reasoning, then reuses the draft through prefix verification and suffix decoding. The method is implemented in vLLM and evaluated mainly on ClassEval across three model families, with additional experiments on HumanEval. SSR achieves 1.23–1.55× end-to-end speedup over standard decoding, with an additional 6–8% gain over suffix decoding on Gemma and Qwen, but no significant gain on GPT-OSS.

Critical strengths
  1. Using partial CoT responsesas speculative drafts is a simple and interesting idea that requires no extra model or training.

  2. The controlled comparison with suffix decoding helps isolate the benefit of the proposed draft. The analysis of draft timing and token overlap provides useful insights into when SSR works and when it does not.

Critical weaknesses
  1. The related work misses several closely related papers. Asynchronous Reasoning (Yakushev et al., 2025) already explores concurrent reasoning and answer generation without training, while Lanham et al. (2023) study how answers from partial CoTs agree with those from complete CoTs. The paper should clarify how SSR differs from these works beyond its verification and suffix-reuse mechanisms.

  2. The improvements over suffix decoding are modest (6–8%), with no significant gain on GPT-OSS. The evaluated models are also not representative of the latest reasoning models, which limits the practical relevance of the results.

  3. The accuracy evaluation is not fully convincing. Prefix verification changes the original sampling distribution, but the paper only evaluates completed and valid outputs. The different sample counts across methods (e.g., 164 vs. 146 on HumanEval) make it difficult to judge whether SSR preserves accuracy.

  4. The evaluation focuses mainly on ClassEval, and draft settings are selected on the evaluation tasks. Generalization to other tasks remains unclear.

Questions for authors
  1. Can the authors provide a controlled comparison with vLLM's stock n-gram decoding? Appendix E reports a stronger speedup on Gemma, but the settings differ.

  2. How does SSR perform under concurrent serving? Does the extra computation from drafting affect throughput or memory usage?

  3. Why do the numbers of evaluated outputs differ across methods in Table 2? How would the accuracy results change if incomplete outputs were counted as failures?

Reviewer_qePwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SSR, a training-free self-speculative decoding method that generates a draft answer conditioned on a partial chain-of-thought (CoT). Once the target model completes its full CoT, the draft tokens are verified against the target model and are also reused as an additional source for suffix decoding. The paper mainly evaluates SSR on ClassEval using Gemma and Qwen models, achieving approximately 6–8% speedup over Suffix Decoding.

Critical strengths
  1. It is intuitive that the final answer can be partially generated without requiring the complete CoT. The authors demonstrate that drafts generated from partial CoTs can already achieve substantial overlap with the final answer.
  2. For tasks involving long CoTs, it makes sense to utilize the relatively low GPU utilization during CoT generation to perform additional computation that accelerates final answer generation.
Critical weaknesses
  1. Lack of evaluation under high concurrency. The paper does not evaluate the performance of SSR under high-concurrency workloads, making it unclear whether the reported speedups can be maintained in realistic serving scenarios.
  2. Limited applicability. The evaluated benchmarks mainly involve relatively long answers, with CoT and answer lengths being roughly comparable. While SSR achieves improvements in such settings, its effectiveness remains unclear in two other common scenarios: when the CoT is relatively short, as in GPT-OSS, or when the CoT is significantly longer than the final answer. The paper lacks sufficient experiments to demonstrate the effectiveness of SSR under these conditions.
  3. Relatively weak baselines. The baseline comparisons are insufficient. The authors should at least include comparisons with stronger retrieval-based speculative decoding methods, such as SAM Decoding.
Questions for authors

Please refer to the weaknesses above.

Reviewer_gKG7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces: SSR, a training free self-speculative decoding strategy, that uses a reasoning model’s own partial-CoT response to accelerate generation of its final response. The motivation is that,, which the paper shows, partial-CoT answers contains much of the final answer. The draft runs in parallel with full reasoning, so its cost is mostly hidden. Once reasoning ends, Draft is then used in 2 ways. First prefix verification against the full-CoT distribution. Then, also as an extra source for suffix decoding along with prompt, CoT and response so far. It is evaluated on ClassEval with Gemma-4-E4B, Qwen3-4B-Thinking and GPT-OSS-20B, and on HumanEval with Gemma

Critical strengths
  • The idea is simple and presented clearly. It presents a nice observation of using the model’s own answer from a partial CoT as a draft. It needs no training. Could be a potentially useful direction to pursue for reasoning models for latency efficiency gains.
  • Table 6 and Table 3 are useful to understand when the draft actually helps. Authors are also honest about the limits, like the GPT-OSS result, the settings being tuned on the test tasks, etc
Critical weaknesses
  • Contribution feels incremental, as in it seems limited to adding one more source on top of existing suffix decoding. More work needs to be done in this direction to have enough contribution for a main conference paper. The gain is also limited and not significant in case of GPT-OSS. It raises the question if SSR only helps models with presence of over-thinking behaviour.
  • It's only evaluated on coding tasks and additionally the task queries have decent overlap with the final response. Like in ClassEval 55-67% of the answer is already present in the prompt (Table 6). It would help to see some tasks where the final output is long and the overlap is low.
Questions for authors
  • Does SSR benefits from over-thinking, as in the model arrives at its final answer midway and keeps reasoning over the same logic, so the midway draft already matches the final answer. Figure 1 also shows overlap flattening after the middle of the CoT.
  • What happens on tasks where the final output is long but the prompt overlap is low? For example, code generation from only a problem description like LiveCodeBench, or long-form writing like reports.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.