acceptodds
Under review as a conference paper at ICLR 2027

Draft Calibration for Speculative Decoding: Accelerating LLM Serving by Matching Draft and Target Distributions

Abstract

Speculative decoding accelerates large language model (LLM) inference by employing a lightweight draft model to propose draft tokens and allowing the target model to verify the new tokens in parallel to improve inference-time compute utilization. The speedup relies on the acceptance probability of draft tokens in target verification, which is directly related to the gap between the target distribution and the draft distribution . In this paper, we study draft calibration of speculative decoding to improve the speedup of speculative decoding, where we employ post-hoc methods to calibrate the draft distribution to mitigate the distribution gap. We observe that though current community open-source draft models may learn the top candidate token correctly, they are generally underconfident compared to the target model, suggesting a consistent distribution mismatch. To mitigate the effect of this mismatch on the acceptance rate of the draft models, we try various post-hoc calibration methods, including standard temperature scaling and token-wise calibration. Our results suggest that a standard temperature scaling applied to the target distribution improves the acceptance length by approximately 10% at no additional cost across different settings. Analysis of more general calibration methods reveals further potential room for improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, temperature scaling, calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JNY5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies post-hoc calibration of draft distributions in speculative decoding. Sharpening the draft with a scalar temperature improves online accepted length across several target–draft pairs. Offline trajectory replay then compares temperature scaling with Dirichlet calibration and token-adaptive heads, showing a gap between diagnostic token oracles and learned calibration policies.

Critical strengths
  1. The simple draft-only intervention has practical value. It changes proposal quality while retaining the target distribution under the rejection-sampling construction, and the online evaluation covers several drafter architectures and task types.
  2. The paper goes beyond reporting a favorable temperature: it investigates more expressive calibrators, reports their limited gains, and quantifies replay–online discrepancies rather than treating replay as exact inference.
Critical weaknesses
  1. The main online metric is accepted length, not measured serving throughput or latency. Continuous batching can change the relationship between accepted tokens and cycle cost, so the paper establishes an acceptance improvement more directly than an actual serving speedup.
  2. The scalar and diagonal calibration optima are chosen on the test set, whereas learned matrices and heads are selected on validation data. They are appropriately described as oracle comparators, but this asymmetric selection weakens a general conclusion that more expressive deployable calibration cannot improve over scalar temperature scaling.
Questions for authors
  1. What throughput or latency gain is observed with the same serving configuration when the draft temperature is fixed using separate calibration requests?
  2. How do scalar temperature, diagonal scaling, and learned heads compare when all deployment choices use the same validation split?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer JNY5 (Part 1/2)

We thank the reviewer for the invaluable feedback and recognition of our paper. For easier checking, we mark the modified text in the paper in blue, in response to the questions and suggestions. Below, we provide our discussion.

W1 & Q1. Needed serving efficiency

W1: The main online metric is accepted length, not measured serving throughput or latency. Continuous batching can change the relationship between accepted tokens and cycle cost, so the paper establishes an acceptance improvement more directly than an actual serving speedup.

Q1: What throughput or latency gain is observed with the same serving configuration when the draft temperature is fixed using separate calibration requests?

Thank you for pointing out the role of continuous batching and the need for measurements with independently selected temperatures. Our initial choice of accepted length was motivated by the negligible inference overhead of temperature scaling and the limited additional gains from more complex methods, including Dirichlet scaling and calibration heads. Accepted length therefore provides a useful measure of the efficiency benefit of draft calibration. However, we agree with the reviewer that continuous batching can indeed affect the relationship between accepted length and serving speed improvement, and that it is worth measuring throughput to provide the most direct evidence of speedup. We have therefore added the following experiments.

We have completed a comparison between speculative decoding with and without draft temperature scaling on the same held-out test split used in the paper's calibration experiments, across all eight tasks. One shared temperature per draft/target model pair is selected on separate validation requests and frozen across tasks and loads: 0.4 for GPT-OSS-120B/EAGLE-3 and GPT-OSS-120B/DFlash, and 0.6 for Qwen3.6-35B-A3B/DSpark. Both conditions use matched prompts, seeds, and server settings with two independent single-GPU replicas. Client concurrency is the total in-flight request limit across them.

We repeat each comparison with three sampling seeds and report mean ± standard error (SE; n=3). Within each seed, accepted length and throughput (output tokens/s) come from the same runs, and each metric is averaged across the eight tasks following the same metric as the paper. The table summarizes these absolute values and paired percentage gains across seeds. Parentheses show gains, with standard errors in percentage points. In the table, AL denotes accepted length, and calibrated means with temperature scaling.

Model / drafterClient concurrencyAL, originalAL, calibrated (gain)Throughput, originalThroughput, calibrated (gain)
GPT-OSS-120B / EAGLE-3163.102 ± 0.0043.432 ± 0.003 (10.64 ± 0.19%)760.9 ± 11.6836.2 ± 8.9 (9.92 ± 1.02%)
GPT-OSS-120B / EAGLE-3643.104 ± 0.0053.436 ± 0.008 (10.71 ± 0.28%)1582.6 ± 25.91756.3 ± 30.3 (11.01 ± 2.03%)
GPT-OSS-120B / EAGLE-31283.103 ± 0.0033.441 ± 0.009 (10.89 ± 0.36%)2141.6 ± 32.52249.7 ± 38.6 (5.13 ± 3.02%)
GPT-OSS-120B / DFlash163.657 ± 0.0093.996 ± 0.011 (9.27 ± 0.45%)1449.5 ± 8.31592.5 ± 6.0 (9.87 ± 0.29%)
GPT-OSS-120B / DFlash643.656 ± 0.0113.994 ± 0.004 (9.25 ± 0.44%)2765.9 ± 26.53034.3 ± 61.8 (9.68 ± 1.29%)
GPT-OSS-120B / DFlash1283.660 ± 0.0113.998 ± 0.015 (9.26 ± 0.64%)3548.1 ± 22.53866.0 ± 66.8 (8.98 ± 2.32%)
Qwen3.6-35B-A3B / DSpark164.049 ± 0.0104.434 ± 0.006 (9.51 ± 0.32%)1174.9 ± 37.51313.9 ± 22.2 (12.04 ± 3.86%)
Qwen3.6-35B-A3B / DSpark644.066 ± 0.0224.425 ± 0.009 (8.82 ± 0.39%)1858.6 ± 37.32265.8 ± 74.1 (22.08 ± 5.64%)
Qwen3.6-35B-A3B / DSpark1284.070 ± 0.0214.430 ± 0.012 (8.84 ± 0.29%)1782.9 ± 33.22130.7 ± 33.7 (19.53 ± 1.15%)

As presented in the table, mean throughput improves by 5.13%–22.08%, alongside mean accepted-length gains of 8.82%–10.89%. The throughput percentage gains are generally consistent with the accepted length in most of the settings, consistent with the negligible overhead of temperature scaling. Larger differences occur for DSpark at concurrencies 64 and 128, and for EAGLE-3 at concurrency 128. We think this gain mismatch could be related to continuous batching, which lets requests enter and finish at different times, and differences in sampled output lengths can change the active batch sizes and context lengths, affecting the effective verification speed. Also, mean request-latency reductions across the three seeds are positive in every condition: 5.60%–12.43% for p50 and 5.84%–16.11% for p95. To conclude, temperature scaling generally still preserves the speedup benefits when comparing throughputs.

Thanks for the suggestion again. We have added the throughput results in Table 1 in Section 5.1, and presented the latency and more details in Appendix B.3.

AuthorsOpenReview ↗

Reply to Reviewer JNY5 (Part 2/2)

W2 & Q2. Validating splits of the results

W2: The scalar and diagonal calibration optima are chosen on the test set, whereas learned matrices and heads are selected on validation data. They are appropriately described as oracle comparators, but this asymmetric selection weakens a general conclusion that more expressive deployable calibration cannot improve over scalar temperature scaling.

Q2: How do scalar temperature, diagonal scaling, and learned heads compare when all deployment choices use the same validation split?

Thank you for this insightful suggestion. The submitted scalar and diagonal results were test-selected oracle comparators. We now supplement them with validation-selected scalar and diagonal results, so all deployment choices use the same validation split and the test oracles remain separately labeled. This selection rule has also been updated in Tables 3 and 4 of the revised paper.

All methods share the whole-request train/validation/test split (1,660/413/2,082 requests per model pair). The table below reports test replay LposL_{\mathrm{pos}}, averaged across eight tasks, after validation-based selection. Note that the scalar here is selected per task.

MethodGPT-OSS / EAGLE-3GPT-OSS / DFlashQwen3.6 / DSpark
Scalar temperature, per task3.50854.07514.4825
Diagonal scaling3.50844.07524.4815
Full matrix3.48244.05634.4192
Temperature head, draft state3.51114.10914.4765
Temperature head, draft + previous target state3.51704.10914.4726
Binary head, draft state3.40314.00454.3281
Binary head, draft + previous target state3.40023.99984.3234

For the scalar baseline, validation-selected temperatures equal the test-selected temperatures on all eight tasks for both DFlash and EAGLE-3 pairs. For Qwen3.6 / DSpark, they agree on five tasks, and the task-mean score changes from 4.4847 to 4.4825, approximately 0.05%. When diagonal Dirichlet scaling is selected using the validation set, it remains essentially tied with scalar scaling. It even slightly underperforms scalar scaling in some cases on the test set. Temperature heads with draft-plus-previous-target input improve over scalar scaling by 0.24% and 0.83% for the GPT-OSS/EAGLE-3 and GPT-OSS/DFlash pairs, respectively; both input variants score slightly lower for DSpark. These results support that scalar scaling captures most of the gains among the methods tested, while leaving room for better adaptive calibration.

We have added the validation-selected results to Tables 3–4 and updated the corresponding selection descriptions. Appendix A.3 gives the grids and training details.

Reviewer_QnbYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies draft calibration for speculative decoding, motivated by the fact that speculative-sampling acceptance depends directly on the distribution gap between the target model pp and draft model qq, with acceptance probability 1−12∥p−q∥11-\frac{1}{2}\|p-q\|_1. The authors observe that several open-source draft models often identify the correct high-probability tokens but are systematically underconfident / flatter than their corresponding target models, creating avoidable distribution mismatch. They therefore investigate whether post-hoc calibration of the draft distribution can improve speculative-decoding acceptance while preserving the standard lossless sampling guarantee. 54678_Draft_Calibration_for_Sp

The paper finds that simple draft-side temperature scaling—using a draft temperature TdT_d different from the fixed target temperature TtT_t—often sharpens the draft distribution and improves accepted length.

It also evaluates more expressive Dirichlet scaling and token-adaptive calibration, finding that these methods do not consistently outperform a global temperature adjustment, although token-wise oracle results suggest additional headroom for future adaptive methods.

Critical strengths
  • The paper connects speculative-decoding acceptance directly to the draft–target distribution gap, 1−12∥p−q∥11-\frac12\|p-q\|_1, and reframes draft calibration as a way to improve acceptance without changing the target distribution, provided the calibrated draft distribution is used consistently in proposal generation and rejection sampling.

  • The paper evaluates nine target–draft pairs, which shows that simple draft-side temperature scaling often improves accepted length by several percent to around 10% without additional model computation.

Critical weaknesses
  • The paper’s main intervention is to apply existing post-hoc calibration techniques—especially standard temperature scaling—to the draft distribution after observing that several draft models are systematically flatter than their targets. The observation is useful, but the core method itself is straightforward and largely established.

  • The main serving claim is not directly validated under a deployable protocol. The primary metric is accepted length, while the title and framing emphasize accelerating LLM serving.

Questions for authors

Can you provide a detailed explanation of your Figure 5, which seems to be your main pipeline?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer QnbY (Part 1/3)

We thank the reviewer for the effort in reading our paper and valuable feedback. For easier checking, we mark the modified text in the paper in blue, in response to the questions and suggestions. We provide our responses to the points below.

W1. Method novelty

The paper’s main intervention is to apply existing post-hoc calibration techniques—especially standard temperature scaling—to the draft distribution after observing that several draft models are systematically flatter than their targets. The observation is useful, but the core method itself is straightforward and largely established.

We thank the reviewer for raising the question about the novelty of the paper. While we agree with the reviewer that temperature scaling is itself well-established, we would like to kindly clarify that our paper goes beyond the observation that temperature scaling can help.

First, rather than simply trying the method and observing that it works, the study is motivated by the observed draft–target distribution gap: draft models can learn the top candidates well, but assign them systematically flatter probabilities (Figures 1–2). At a fixed context, acceptance equals 1−12∥p−q∥11-\frac12\|p-q\|_1 for target distribution pp and draft distribution qq, so any calibration that reduces this distance improves acceptance. This motivates using calibration to correct the observed mismatch.

Second, with this motivation for draft calibration, the final recommended intervention is temperature scaling, but the recommendation stems from thorough comparisons with more general methods, including Dirichlet scaling and token-adaptive heads. Our offline trajectory replay makes extensive searches practical, including over 1.6 million diagonal Dirichlet scaling configurations, to estimate the best performance within the tested families and search grids on cached contexts. These comparisons show that scalar scaling captures most of the gains among the methods tested. We view the simplicity of this recommended method as a practical benefit supported by this analysis, rather than a lack of novelty.

Third, we also show further room for improvement with token-adaptive calibration, suggesting a potentially more powerful intervention than scalar temperature scaling. Target-informed token-wise oracles substantially outperform the best per-task scalar temperature on the same replay contexts (Table 4). Our current post-hoc heads recover only a small part of this potential, suggesting better token-adaptive calibration as a promising direction for further gains beyond a shared temperature.

Our study therefore identifies a useful draft–target probability mismatch and provides a practical way to correct it. The serving measurements below further demonstrate 5.13%–22.08% mean throughput gains with validation-selected temperatures in the tested serving conditions.

AuthorsOpenReview ↗

Reply to Reviewer QnbY (Part 2/3)

W2. Serving efficiency metric

The main serving claim is not directly validated under a deployable protocol. The primary metric is accepted length, while the title and framing emphasize accelerating LLM serving.

Thanks for mentioning the point about measuring acceleration on LLM serving. First, we would like to briefly explain why we would take accepted length as the measurement. As mentioned in Section 5.1, we take accepted length as the major metric because the final recommended method, temperature scaling, adds almost no model computation to the original pipeline. Therefore, accepted length improvement can imply speedup improvement for temperature scaling, which also directly reflects the draft–target distribution overlap that the calibration methods aim to improve. For Dirichlet scaling and calibration head methods, which do introduce additional computation, accepted length comparison is valid in suggesting that they don't consistently outperform temperature scaling.

But still, we fully agree with the reviewer that considering more direct evidence of serving efficiency under a deployable protocol would be meaningful. We have therefore added the following measurements of throughput and latency in serving, using temperatures selected on separate validation requests.

We have completed a comparison between speculative decoding with and without draft temperature scaling on the same held-out test split used in the paper's experiments in Section 5.2, across all eight tasks. One shared temperature per draft/target model pair is selected on separate validation requests and frozen across tasks and loads: 0.4 for GPT-OSS-120B/EAGLE-3 and GPT-OSS-120B/DFlash, and 0.6 for Qwen3.6-35B-A3B/DSpark. Both conditions use matched prompts, seeds, and server settings with two independent single-GPU replicas. Client concurrency is the total in-flight request limit across them.

We repeat each comparison with three sampling seeds and report mean ± standard error (SE; n=3). Within each seed, accepted length and throughput (output tokens/s) come from the same runs, and each metric is averaged across the eight tasks following the same metric as the paper. The table summarizes these absolute values and paired percentage gains across seeds. Parentheses show gains, with standard errors in percentage points. In the table, AL denotes accepted length, and calibrated means with temperature scaling.

Model / drafterClient concurrencyAL, originalAL, calibrated (gain)Throughput, originalThroughput, calibrated (gain)
GPT-OSS-120B / EAGLE-3163.102 ± 0.0043.432 ± 0.003 (10.64 ± 0.19%)760.9 ± 11.6836.2 ± 8.9 (9.92 ± 1.02%)
GPT-OSS-120B / EAGLE-3643.104 ± 0.0053.436 ± 0.008 (10.71 ± 0.28%)1582.6 ± 25.91756.3 ± 30.3 (11.01 ± 2.03%)
GPT-OSS-120B / EAGLE-31283.103 ± 0.0033.441 ± 0.009 (10.89 ± 0.36%)2141.6 ± 32.52249.7 ± 38.6 (5.13 ± 3.02%)
GPT-OSS-120B / DFlash163.657 ± 0.0093.996 ± 0.011 (9.27 ± 0.45%)1449.5 ± 8.31592.5 ± 6.0 (9.87 ± 0.29%)
GPT-OSS-120B / DFlash643.656 ± 0.0113.994 ± 0.004 (9.25 ± 0.44%)2765.9 ± 26.53034.3 ± 61.8 (9.68 ± 1.29%)
GPT-OSS-120B / DFlash1283.660 ± 0.0113.998 ± 0.015 (9.26 ± 0.64%)3548.1 ± 22.53866.0 ± 66.8 (8.98 ± 2.32%)
Qwen3.6-35B-A3B / DSpark164.049 ± 0.0104.434 ± 0.006 (9.51 ± 0.32%)1174.9 ± 37.51313.9 ± 22.2 (12.04 ± 3.86%)
Qwen3.6-35B-A3B / DSpark644.066 ± 0.0224.425 ± 0.009 (8.82 ± 0.39%)1858.6 ± 37.32265.8 ± 74.1 (22.08 ± 5.64%)
Qwen3.6-35B-A3B / DSpark1284.070 ± 0.0214.430 ± 0.012 (8.84 ± 0.29%)1782.9 ± 33.22130.7 ± 33.7 (19.53 ± 1.15%)

As presented in the table, mean throughput improves by 5.13%–22.08%, alongside mean accepted-length gains of 8.82%–10.89%. The throughput percentage gains are generally consistent with the accepted length in most of the settings, consistent with the negligible overhead of temperature scaling. Larger differences occur for DSpark at concurrencies 64 and 128, and for EAGLE-3 at concurrency 128. We think this gain mismatch could be related to continuous batching, which lets requests enter and finish at different times, and differences in sampled output lengths can change the active batch sizes and context lengths, affecting the effective verification speed. Also, mean request-latency reductions across the three seeds are positive in every condition: 5.60%–12.43% for p50 and 5.84%–16.11% for p95. To conclude, temperature scaling generally still preserves the speedup benefits when comparing throughputs.

Thanks for the suggestion again. We have added the throughput results in Table 1 in Section 5.1, and presented the latency and more details in Appendix B.3.

AuthorsOpenReview ↗

Reply to Reviewer QnbY (Part 3/3)

Q. A detailed explanation of Figure 5

Can you provide a detailed explanation of your Figure 5, which seems to be your main pipeline?

Thanks for the question. We would like to first kindly clarify that Figure 5 illustrates the offline trajectory replay pipeline used to examine more complex calibration methods, including Dirichlet calibration and trained calibration heads, and our main experiments for evaluating the effectiveness of temperature scaling are simply normal online inference with speculative decoding, not under this pipeline.

As also included in Appendix B.2, the replay procedure works as follows:

  1. Generate reference trajectories at Td=Tt=1T_d=T_t=1 and save draft and target final hidden states at positions where the preceding draft proposals were accepted.
  2. Reconstruct the draft and target distributions using their output heads.
  3. Apply a candidate calibration to the draft logits and compute its overlap with the target, A=∑xmin⁡(p(x),qcal(x))A=\sum_x\min(p(x),q_{\mathrm{cal}}(x)).
  4. Aggregate these overlaps into LposL_{\mathrm{pos}} to compare calibration candidates on the same reference contexts.

We have clarified the use of fixed reference contexts in Section 5.2.1. Appendix B.2 also compares replay with fresh online decoding, with mean absolute relative errors of 1.83%–2.64% across the three model pairs.

Our goal for introducing this pipeline is to make searches over large spaces, such as the diagonal calibration grid, practical and estimate the best performance within the tested calibration families and search grids on cached contexts. For the token-oracle analysis, we additionally choose the best temperature at each cached position using the target distribution. This reveals the remaining headroom for token-adaptive calibration within these grids and helps assess how much of that gain learned heads recover.

Reviewer_YqGjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies post-hoc calibration of draft distributions for speculative decoding. It observes that many tested drafters produce flatter distributions than their targets, and shows that lowering the draft temperature improves accepted length by approximately 3–11% across the main evaluated configurations. More elaborate calibration methods are shown to provide limited additional gains.

Critical strengths

S1) The proposed intervention is simple, inexpensive, and preserves the target sampling distribution. Improving acceptance through a scalar parameter, without changing model weights, could be useful in existing serving systems.

S2) The main temperature-scaling results are supported by online decoding experiments across nine target–draft pairs and eight workloads. The relatively broad temperature ranges that perform well suggest that the gains do not require especially precise tuning.

S3) The paper goes beyond reporting a successful temperature adjustment by examining more expressive calibration methods and their limitations.

Critical weaknesses

W1) The practical speed benefit needs clarification and additional results. The paper reports accepted length rather than end-to-end latency or throughput. Although changing the draft temperature should introduce little computational overhead, the claim of improved serving efficiency would be stronger with timing measurements under representative serving loads.

W2) The generality of the underconfidence finding needs qualification. Appendix B.1 reports gains of only 2.05%, 0.10%, and 0% for three additional DeepSeek-trained draft checkpoints. These results are important for understanding the scope of the contribution and should appear more prominently in the main discussion. What differences between checkpoints explain when temperature calibration helps?

Questions for authors

Two questions related to the two weaknesses.

W1) The practical speed benefit needs clarification and additional results. The paper reports accepted length rather than end-to-end latency or throughput. Although changing the draft temperature should introduce little computational overhead, the claim of improved serving efficiency would be stronger with timing measurements under representative serving loads.

W2) The generality of the underconfidence finding needs qualification. Appendix B.1 reports gains of only 2.05%, 0.10%, and 0% for three additional DeepSeek-trained draft checkpoints. These results are important for understanding the scope of the contribution and should appear more prominently in the main discussion. What differences between checkpoints explain when temperature calibration helps?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer YqGj (Part 1/2)

We appreciate the reviewer for the insightful comments and recognition of our paper. For easier checking, we mark the modified text in the paper in blue, in response to the questions and suggestions. Below, we provide our responses.

W1 & Q1. Throughput results.

The practical speed benefit needs clarification and additional results. The paper reports accepted length rather than end-to-end latency or throughput. Although changing the draft temperature should introduce little computational overhead, the claim of improved serving efficiency would be stronger with timing measurements under representative serving loads.

Thanks for raising the point on throughput evaluation. Our initial choice of accepted length was motivated by the negligible inference overhead of temperature scaling, as the reviewer also notes. It directly reflects the acceptance benefit of calibration, and more complex methods offer limited additional gains. We still agree and appreciate the reviewer's comment that measured throughput and latency provide more direct evidence of serving efficiency, so we have added timing measurements at three client concurrencies.

We have completed a comparison between speculative decoding with and without draft temperature scaling on the same held-out test split used in the paper's calibration experiments, across all eight tasks. One shared temperature per draft/target model pair is selected on separate validation requests and frozen across tasks and loads: 0.4 for GPT-OSS-120B/EAGLE-3 and GPT-OSS-120B/DFlash, and 0.6 for Qwen3.6-35B-A3B/DSpark. Both conditions use matched prompts, seeds, and server settings with two independent single-GPU replicas. Client concurrency is the total in-flight request limit across them.

We repeat each comparison with three sampling seeds and report mean ± standard error (SE; n=3). Within each seed, accepted length and throughput (output tokens/s) come from the same runs, and each metric is averaged across the eight tasks following the same metric as the paper. The table summarizes these absolute values and paired percentage gains across seeds. Parentheses show gains, with standard errors in percentage points. In the table, AL denotes accepted length, and calibrated means with temperature scaling.

Model / drafterClient concurrencyAL, originalAL, calibrated (gain)Throughput, originalThroughput, calibrated (gain)
GPT-OSS-120B / EAGLE-3163.102 ± 0.0043.432 ± 0.003 (10.64 ± 0.19%)760.9 ± 11.6836.2 ± 8.9 (9.92 ± 1.02%)
GPT-OSS-120B / EAGLE-3643.104 ± 0.0053.436 ± 0.008 (10.71 ± 0.28%)1582.6 ± 25.91756.3 ± 30.3 (11.01 ± 2.03%)
GPT-OSS-120B / EAGLE-31283.103 ± 0.0033.441 ± 0.009 (10.89 ± 0.36%)2141.6 ± 32.52249.7 ± 38.6 (5.13 ± 3.02%)
GPT-OSS-120B / DFlash163.657 ± 0.0093.996 ± 0.011 (9.27 ± 0.45%)1449.5 ± 8.31592.5 ± 6.0 (9.87 ± 0.29%)
GPT-OSS-120B / DFlash643.656 ± 0.0113.994 ± 0.004 (9.25 ± 0.44%)2765.9 ± 26.53034.3 ± 61.8 (9.68 ± 1.29%)
GPT-OSS-120B / DFlash1283.660 ± 0.0113.998 ± 0.015 (9.26 ± 0.64%)3548.1 ± 22.53866.0 ± 66.8 (8.98 ± 2.32%)
Qwen3.6-35B-A3B / DSpark164.049 ± 0.0104.434 ± 0.006 (9.51 ± 0.32%)1174.9 ± 37.51313.9 ± 22.2 (12.04 ± 3.86%)
Qwen3.6-35B-A3B / DSpark644.066 ± 0.0224.425 ± 0.009 (8.82 ± 0.39%)1858.6 ± 37.32265.8 ± 74.1 (22.08 ± 5.64%)
Qwen3.6-35B-A3B / DSpark1284.070 ± 0.0214.430 ± 0.012 (8.84 ± 0.29%)1782.9 ± 33.22130.7 ± 33.7 (19.53 ± 1.15%)

As presented in the table, mean throughput improves by 5.13%–22.08%, alongside mean accepted-length gains of 8.82%–10.89%. The throughput percentage gains are generally consistent with the accepted length in most of the settings, consistent with the negligible overhead of temperature scaling. Larger differences occur for DSpark at concurrencies 64 and 128, and for EAGLE-3 at concurrency 128. We think this gain mismatch could be related to continuous batching, which lets requests enter and finish at different times, and differences in sampled output lengths can change the active batch sizes and context lengths, affecting the effective verification speed. Also, mean request-latency reductions across the three seeds are positive in every condition: 5.60%–12.43% for p50 and 5.84%–16.11% for p95. To conclude, temperature scaling generally still preserves the speedup benefits when comparing throughputs.

Thanks for the suggestion again. We have added the throughput results in Table 1 in Section 5.1, and presented the latency and more details in Appendix B.3.

AuthorsOpenReview ↗

Reply to Reviewer YqGj (Part 2/2)

W2 & Q2. The generality of the underconfidence finding needs qualification.

The generality of the underconfidence finding needs qualification. Appendix B.1 reports gains of only 2.05%, 0.10%, and 0% for three additional DeepSeek-trained draft checkpoints. These results are important for understanding the scope of the contribution and should appear more prominently in the main discussion. What differences between checkpoints explain when temperature calibration helps?

Thanks for the constructive suggestion on the results for DeepSeek-trained models. We have added Table 2 and a brief discussion in Section 5.1, with the full temperature sweeps and distribution diagnostics in Appendix B.1.

To examine when temperature scaling helps, we applied the Figure 2 diagnostics to all three DeepSeek checkpoints and compared the draft models below. The top-1 probability gap is target minus draft, conditioned on argmax agreement; the entropy gap is draft minus target, averaged over all eligible positions.

The final column reports gains in equal-task mean accepted length over Td=1T_d=1, using the best shared temperature across eight tasks from the original online sweeps (Figures 3 and 7).

Draft algorithmTarget / checkpoint groupTop-1 probability gap (percentage points)Entropy gap (nats)Accepted length gain
EAGLE-3GPT-OSS-120B (Figure 2)11.940.84211.47%
Qwen3-14B (DeepSeek)5.650.4432.05%
DFlashGPT-OSS-120B (Figure 2)9.860.64110.11%
Qwen3-14B (DeepSeek)0.930.1770.10%
DSparkQwen3.6-35B-A3B (Figure 2)7.350.7448.82%
Qwen3-14B (DeepSeek)0.180.0750%

We can observe that these DeepSeek-trained checkpoints have smaller probability and entropy gaps compared to other checkpoints even with the same draft model algorithm, consistent with their limited gains from temperature scaling. This also matches our motivation for employing draft calibration: it helps correct the consistent distribution gap between draft and target models. When the probability mismatch is small, as in these models, temperature scaling offers little additional benefit. One possible reason for the distribution gap difference between DeepSeek-trained models and others is the training recipe, e.g., training contexts that better match inference-time draft trajectories, training lengths, and the loss objective could make the draft model better calibrated. The target model may also matter, since all three checkpoints share Qwen3-14B as the target.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.