acceptodds
Withdrawn from ICLR 2027

CSD: Confidence- and Stability-Guided Speculative Decoding in Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding has emerged as a promising technique to accelerate large language model inference by leveraging a lightweight draft model to propose candidate tokens for parallel verification by a target model. Although recent methods have explored adaptive draft-length control, efficiently estimating when the draft model can safely speculate more aggressively remains challenging. In this paper, we propose Confidence- and Stability-guided speculative Decoding (CSD), a training-free method that relies only on draft-model runtime signals, including token-level prediction confidence and recent local stability. CSD adopts a lightweight two-level framework to dynamically adjust the drafting boundary during decoding. Experiments across multiple model pairs and datasets show that CSD can be integrated into both standard and parallel speculative decoding without modifying the target verification rule and achieves a – speedup over autoregressive decoding. CSD achieves competitive or better throughput than strong fixed-length and parallel speculative decoding baselines across the evaluated settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Efficient LLM Inference, Dynamic Decoding Strategy

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qnnaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to choose the number of draft tokens in speculative decoding, arguing that a fixed length cannot adapt to changing prediction confidence.

It proposes CSD, which uses the draft model’s confidence and recent changes in probability margins to choose an initial draft length, then permits a short extension when confidence and margins remain high.

Experiments report speedups over autoregressive decoding and improvements over speculative decoding baselines.

Critical strengths
  1. A simple method with practical integration potential.

CSD uses probability statistics already available during drafting, requires no additional training, and retains the underlying verification procedure.

  1. The experiments include meaningful component ablations.

Table 2 shows that round-level control and late extension each improve throughput, with their combination performing best across four datasets.

Critical weaknesses

The manuscript’s presentation makes it difficult to connect the motivation, decision rules, and experimental evidence.

  1. Figure 1 occupies substantial space but leaves its “difficulty score” undefined and does not clarify whether the curve is empirical or schematic (lines 54–107). More substantively, “stability” denotes historical margin variation in round-level control but a current margin exceeding its historical median in late extension, without explaining the distinction (lines 197–262).
  2. The relationship and update rules of the two history structures are unclear, and the serial and parallel descriptions do not consistently explain the signals available when draft length is selected (lines 246–293).
  3. Tables also fail to distinguish planned from actual draft lengths, while Tables 3 and 5 report different throughputs and reverse fixed-length rankings without explaining configuration differences.
  4. Repeated citation-formatting errors, an uncited reference to SpecDec++, and disruptive figure/table placement further impede reading. Together, these issues prevent the reader from confidently reconstructing the method and interpreting the evidence supporting its claims.
Questions for authors
  1. Evidence for the motivation: Is Figure 1 an empirical plot or a schematic illustration? What exactly is its “difficulty score,” and how does it relate to the confidence and stability statistics used by CSD? For the analysis in lines 99–107, how was confidence controlled when assessing the additional information provided by stability?

  2. Algorithm specification: Round-level control measures variation in historical margins, whereas late extension compares the current margin against its historical median. What justifies interpreting the latter as stability? Please clarify the relationship, initialization, and update rules of (B_t) and (H_t), and which current confidence value is available when parallel decoding selects its planned length.

Reviewer_dH9ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSD, a training-free method that dynamically adjusts speculative decoding draft lengths using draft-model confidence and recent prediction stability. It combines round-level length control with token-level late extension and integrates into both standard and parallel speculative decoding without changing the target-model verification rule.

Critical strengths
  • I love the motivation which is clear and supported by empirical evidence. Token acceptance statistics and ablation studies demonstrate the relevance of confidence and local stability to adaptive draft-length control.
  • The method is simple and controllable. It uses readily available draft-model statistics without additional training, while explicit length bounds and a capped extension budget make its behavior easy to configure.
  • The paper provides useful mechanistic analysis beyond throughput, examining acceptance rates, average accepted lengths, and draft/target forward-pass counts to explain the sources of acceleration.
Critical weaknesses
  • The advantage over sufficiently tuned baselines remains insufficiently established. PEARL uses its default base window, while the fixed-length sweep covers only one model–dataset combination. In that setting, CSD improves throughput over the best fixed length by only approximately 0.91% (36.45 vs. 36.12 tokens/s), without repeated-run variability or confidence intervals. AdaEDL, a directly relevant adaptive baseline, is also absent. Broader comparisons under comparable tuning budgets, uncertainty estimates, and an AdaEDL comparison would strengthen the empirical claims.
  • The configuration differences between Tables 3 and 5 are unclear. Both use DeepSeek-Coder-6.7B/33B on CNN/DailyMail, yet report CSD throughput of 28.31 and 36.45 tokens/s, respectively; the corresponding fixed-γ=6\gamma=6 results are 24.38 and 36.12 tokens/s. These differences may reflect different execution frameworks or settings, but the paper does not clearly explain them. Please specify the standard/parallel decoding setup, evaluation samples, decoding parameters, and timing protocol for each table.
Questions for authors

I hope the authors could address the weakness above.

Reviewer_F3ZxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Confidence- and Stability-guided Speculative Decoding (CSD), a training-free framework for adaptive draft-length control in speculative decoding. It combines draft-model prediction confidence with recent local stability, then extends the draft token by token when these signals remain reliable. CSD can be used in both standard and parallel speculative decoding while keeping the target verification rule unchanged. Experiments across several model pairs and benchmarks report higher throughput than fixed-length drafting and existing adaptive decoding methods.

Critical strengths
  1. Practical and lightweight design. CSD uses statistics already available from the draft model to adjust draft length. It requires no additional training and leaves target verification unchanged. Support for both standard and parallel speculative decoding makes it straightforward to incorporate into existing systems.

  2. Evaluation across models and tasks. The experiments cover multiple draft-target model pairs and generation tasks. The ablations in Tables 2–4 and the sensitivity analyses support the contribution of both confidence and local stability to decoding efficiency. Table 3 also reports acceptance rates and forward passes, helping explain where the throughput gains come from.

Critical weaknesses
  1. Limited novelty and a small gain over the best tested fixed length. CSD combines confidence-based length adjustment, historical stability statistics, and threshold-based late extension. The conceptual advance over existing adaptive drafting policies appears limited. Table 5 reports only a 0.9% throughput gain over the best tested fixed draft length (36.45 vs. 36.12 tokens/s). The larger gains over PEARL and other baselines are harder to interpret without knowing how their drafting configurations were chosen. It remains unclear how much benefit comes from adaptation once the baselines are tuned.

  2. Throughput differences across tables need explanation. Tables 3 and 5 report different throughput values for CSD (28.31 vs. 36.45 tokens/s) and fixed γ=6γ = 6 (24.38 vs. 36.12 tokens/s), although both describe the DeepSeek-Coder-6.7B/33B pair and CNN/DailyMail. Appendix Table 6 also reports 34.36 tokens/s for the controller without late extension, compared with 27.15 tokens/s for the corresponding configuration in Table 3. The paper needs to specify which settings or timing protocols differ so that these efficiency comparisons can be assessed reliably.

Questions for authors
  1. Throughput differences. Why do Tables 3 and 5 report different throughput values for the same model pair and dataset? How does the no-late-extension configuration in Table 3 relate to $ \alpha = 1, \beta = 5 $ in Appendix Table 6? Please specify any differences in execution mode, generation length, timing protocol, or other relevant settings.

  2. Benefits over independently tuned baselines. Table 5 shows a 0.9% gain over the best tested fixed draft length, while Table 1 reports larger gains over existing decoding methods. Were PEARL and the fixed-length baselines tuned independently under comparable settings? Using existing results or a limited additional comparison, how much improvement remains when each method uses its best tested drafting configuration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.