- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies post-hoc calibration of draft distributions in speculative decoding. Sharpening the draft with a scalar temperature improves online accepted length across several target–draft pairs. Offline trajectory replay then compares temperature scaling with Dirichlet calibration and token-adaptive heads, showing a gap between diagnostic token oracles and learned calibration policies.
Critical strengths
- The simple draft-only intervention has practical value. It changes proposal quality while retaining the target distribution under the rejection-sampling construction, and the online evaluation covers several drafter architectures and task types.
- The paper goes beyond reporting a favorable temperature: it investigates more expressive calibrators, reports their limited gains, and quantifies replay–online discrepancies rather than treating replay as exact inference.
Critical weaknesses
- The main online metric is accepted length, not measured serving throughput or latency. Continuous batching can change the relationship between accepted tokens and cycle cost, so the paper establishes an acceptance improvement more directly than an actual serving speedup.
- The scalar and diagonal calibration optima are chosen on the test set, whereas learned matrices and heads are selected on validation data. They are appropriately described as oracle comparators, but this asymmetric selection weakens a general conclusion that more expressive deployable calibration cannot improve over scalar temperature scaling.
Questions for authors
- What throughput or latency gain is observed with the same serving configuration when the draft temperature is fixed using separate calibration requests?
- How do scalar temperature, diagonal scaling, and learned heads compare when all deployment choices use the same validation split?
2 comments
Reply to Reviewer JNY5 (Part 2/2)
W2 & Q2. Validating splits of the results
W2: The scalar and diagonal calibration optima are chosen on the test set, whereas learned matrices and heads are selected on validation data. They are appropriately described as oracle comparators, but this asymmetric selection weakens a general conclusion that more expressive deployable calibration cannot improve over scalar temperature scaling.
Q2: How do scalar temperature, diagonal scaling, and learned heads compare when all deployment choices use the same validation split?
Thank you for this insightful suggestion. The submitted scalar and diagonal results were test-selected oracle comparators. We now supplement them with validation-selected scalar and diagonal results, so all deployment choices use the same validation split and the test oracles remain separately labeled. This selection rule has also been updated in Tables 3 and 4 of the revised paper.
All methods share the whole-request train/validation/test split (1,660/413/2,082 requests per model pair). The table below reports test replay , averaged across eight tasks, after validation-based selection. Note that the scalar here is selected per task.
| Method | GPT-OSS / EAGLE-3 | GPT-OSS / DFlash | Qwen3.6 / DSpark |
|---|---|---|---|
| Scalar temperature, per task | 3.5085 | 4.0751 | 4.4825 |
| Diagonal scaling | 3.5084 | 4.0752 | 4.4815 |
| Full matrix | 3.4824 | 4.0563 | 4.4192 |
| Temperature head, draft state | 3.5111 | 4.1091 | 4.4765 |
| Temperature head, draft + previous target state | 3.5170 | 4.1091 | 4.4726 |
| Binary head, draft state | 3.4031 | 4.0045 | 4.3281 |
| Binary head, draft + previous target state | 3.4002 | 3.9998 | 4.3234 |
For the scalar baseline, validation-selected temperatures equal the test-selected temperatures on all eight tasks for both DFlash and EAGLE-3 pairs. For Qwen3.6 / DSpark, they agree on five tasks, and the task-mean score changes from 4.4847 to 4.4825, approximately 0.05%. When diagonal Dirichlet scaling is selected using the validation set, it remains essentially tied with scalar scaling. It even slightly underperforms scalar scaling in some cases on the test set. Temperature heads with draft-plus-previous-target input improve over scalar scaling by 0.24% and 0.83% for the GPT-OSS/EAGLE-3 and GPT-OSS/DFlash pairs, respectively; both input variants score slightly lower for DSpark. These results support that scalar scaling captures most of the gains among the methods tested, while leaving room for better adaptive calibration.
We have added the validation-selected results to Tables 3–4 and updated the corresponding selection descriptions. Appendix A.3 gives the grids and training details.
Reply to Reviewer JNY5 (Part 1/2)
We thank the reviewer for the invaluable feedback and recognition of our paper. For easier checking, we mark the modified text in the paper in blue, in response to the questions and suggestions. Below, we provide our discussion.
W1 & Q1. Needed serving efficiency
Thank you for pointing out the role of continuous batching and the need for measurements with independently selected temperatures. Our initial choice of accepted length was motivated by the negligible inference overhead of temperature scaling and the limited additional gains from more complex methods, including Dirichlet scaling and calibration heads. Accepted length therefore provides a useful measure of the efficiency benefit of draft calibration. However, we agree with the reviewer that continuous batching can indeed affect the relationship between accepted length and serving speed improvement, and that it is worth measuring throughput to provide the most direct evidence of speedup. We have therefore added the following experiments.
We have completed a comparison between speculative decoding with and without draft temperature scaling on the same held-out test split used in the paper's calibration experiments, across all eight tasks. One shared temperature per draft/target model pair is selected on separate validation requests and frozen across tasks and loads: 0.4 for GPT-OSS-120B/EAGLE-3 and GPT-OSS-120B/DFlash, and 0.6 for Qwen3.6-35B-A3B/DSpark. Both conditions use matched prompts, seeds, and server settings with two independent single-GPU replicas. Client concurrency is the total in-flight request limit across them.
We repeat each comparison with three sampling seeds and report mean ± standard error (SE; n=3). Within each seed, accepted length and throughput (output tokens/s) come from the same runs, and each metric is averaged across the eight tasks following the same metric as the paper. The table summarizes these absolute values and paired percentage gains across seeds. Parentheses show gains, with standard errors in percentage points. In the table, AL denotes accepted length, and calibrated means with temperature scaling.
As presented in the table, mean throughput improves by 5.13%–22.08%, alongside mean accepted-length gains of 8.82%–10.89%. The throughput percentage gains are generally consistent with the accepted length in most of the settings, consistent with the negligible overhead of temperature scaling. Larger differences occur for DSpark at concurrencies 64 and 128, and for EAGLE-3 at concurrency 128. We think this gain mismatch could be related to continuous batching, which lets requests enter and finish at different times, and differences in sampled output lengths can change the active batch sizes and context lengths, affecting the effective verification speed. Also, mean request-latency reductions across the three seeds are positive in every condition: 5.60%–12.43% for p50 and 5.84%–16.11% for p95. To conclude, temperature scaling generally still preserves the speedup benefits when comparing throughputs.
Thanks for the suggestion again. We have added the throughput results in Table 1 in Section 5.1, and presented the latency and more details in Appendix B.3.