- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents TraceDraft, a lightweight online adaptive framework for speculative decoding. TraceDraft adapts the draft model mid-generation via token-level supervision from the target model, without waiting for fully completed responses. To reduce training overhead while retaining KV cache reuse, the framework only updates the draft model’s output normalization layer and prediction head. It supports Reset and Persistent modes for single-conversation and cross-domain multi-request scenarios. Experiments demonstrate that TraceDraft improves accepted sequence length and end-to-end throughput across eight benchmarks on DeepSeek-R1-Distill-Llama-8B and Vicuna-13B-v1.3.
Critical strengths
- The adaptation mechanism is lightweight. TraceDraft only updates the output normalization layer and prediction head while keeping the draft backbone frozen, preserving existing KV caches. The updated parameters are immediately applied to the remaining response.
Critical weaknesses
- This paper conducts experiments only under greedy sampling with batch size = 1 and synchronous updates. It does not validate the performance under high-concurrency, batched inference or stochastic sampling settings.
Questions for authors
- Can the authors provide end-to-end results under more realistic serving conditions, including interleaved mixed-domain request streams, higher concurrency, larger batch sizes, and stochastic sampling? Such experiments would help assess whether the reported gains generalize beyond the current evaluation settings.
5 comments
Response to Q1 and W1
Dear reviewer, thank you for your constructive suggestion to evaluate TraceDraft under more realistic serving conditions. Following your advice, we have added experiments with interleaved mixed-domain requests, larger batch size, increased concurrency, and stochastic sampling.
For each serving configuration, we construct a mixed-domain stream of 120 questions sampled from LongBench-Write, AIME-2024, AIME-2025, PubMedQA, and TheoremQA, and shuffle their order. The subsets and ordering vary across configurations, while each baseline–Persistent pair processes exactly the same requests in the same order.
We evaluate B=2, C=2 and B=2, C=4, where B is the maximum verification batch size and C is the maximum number of in-flight conversations. Both configurations are tested with greedy decoding (temperature=0, top-p=1) and stochastic sampling (temperature=0.7, top-p=0.9). We use DeepSeek-R1-Distill-Llama-8B with its matched EAGLE-3 draft. Other settings remain fixed. Persistent uses a learning rate of 10⁻⁶, anchor coefficient 0.01, and one adaptation step after five verification rounds in each eligible turn.
Table 1 reports accepted length and domain-specific throughput speedup relative to the matched fixed EAGLE-3 baseline.
Table 1. Accepted length and throughput speedup under batched, concurrent generation (greedy and sampling).
| Setting (B=2) | Dataset | Baseline avg. length | Persistent avg. length | Speedup |
|---|---|---|---|---|
| C=2, Greedy | LongBench-Write | 1.26 | 1.47 | 1.18× |
| C=2, Greedy | AIME-2024 | 2.01 | 2.18 | 1.05× |
| C=2, Greedy | AIME-2025 | 2.12 | 2.21 | 1.02× |
| C=2, Greedy | PubMedQA | 3.67 | 3.84 | 1.02× |
| C=2, Greedy | TheoremQA | 2.32 | 2.47 | 1.06× |
| C=2, Sampling | LongBench-Write | 1.53 | 1.71 | 1.15× |
| C=2, Sampling | AIME-2024 | 1.82 | 1.92 | 1.04× |
| C=2, Sampling | AIME-2025 | 1.87 | 2.06 | 1.09× |
| C=2, Sampling | PubMedQA | 3.57 | 3.79 | 1.07× |
| C=2, Sampling | TheoremQA | 2.17 | 2.28 | 1.03× |
| C=4, Greedy | LongBench-Write | 1.31 | 1.59 | 1.23× |
| C=4, Greedy | AIME-2024 | 2.10 | 2.23 | 1.09× |
| C=4, Greedy | AIME-2025 | 1.98 | 2.13 | 1.07× |
| C=4, Greedy | PubMedQA | 3.67 | 3.91 | 1.09× |
| C=4, Greedy | TheoremQA | 2.09 | 2.31 | 1.08× |
| C=4, Sampling | LongBench-Write | 1.39 | 1.50 | 1.10× |
| C=4, Sampling | AIME-2024 | 1.82 | 1.89 | 1.06× |
| C=4, Sampling | AIME-2025 | 1.76 | 1.83 | 1.02× |
| C=4, Sampling | PubMedQA | 3.60 | 3.70 | 1.03× |
| C=4, Sampling | TheoremQA | 2.15 | 2.25 | 1.07× |
As is shown in the results above, TraceDraft improves accepted length and throughput in all 20 workload–configuration comparisons, with speedups ranging from 1.02× to 1.23×. These gains are consistent in direction and broadly comparable in magnitude to those reported in the paper. In particular, positive gains persist under stochastic sampling and when requests from different domains are interleaved.
We also record the actual verification batch sizes and in-flight conversation counts to verify the execution conditions. Table 2 reports these measurements.
Table 2. Observed batch size and concurrency.
| Setting (B=2) | Method | Mean actual batch | Max actual batch | Mean in-flight | Peak in-flight |
|---|---|---|---|---|---|
| C=2, Greedy | Baseline | 1.969 | 2 | 1.969 | 2 |
| C=2, Greedy | Persistent | 1.977 | 2 | 1.977 | 2 |
| C=2, Sampling | Baseline | 1.970 | 2 | 1.970 | 2 |
| C=2, Sampling | Persistent | 1.984 | 2 | 1.984 | 2 |
| C=4, Greedy | Baseline | 1.953 | 2 | 3.853 | 4 |
| C=4, Greedy | Persistent | 1.981 | 2 | 3.928 | 4 |
| C=4, Sampling | Baseline | 1.989 | 2 | 3.945 | 4 |
| C=4, Sampling | Persistent | 1.990 | 2 | 3.959 | 4 |
As is demonstrated, the observed verification batches remain close to two requests, and the mean in-flight counts reach 3.853–3.959 when C=4. These measurements confirm sustained batched execution with multiple concurrent conversations.
Together, these experiments provide evidence that TraceDraft’s benefits extend beyond the original single-request, greedy evaluation. The shared adaptation configuration retains gains under the tested mixed-domain, batched, concurrent, and stochastic settings. We sincerely thank you again for suggesting this evaluation, which strengthens the evidence for TraceDraft’s practical applicability.
Thank you for the additional experiments, which partially address my concern. However, all new experiments use a verification batch size of B=2. My concern about scaling to larger batches therefore remains. Experiments with larger verification batches would provide more convincing evidence. I maintain my current score.
Thank you for taking time to review our paper and for recognizing the benefit of restricting adaptation to the output layers. We appreciate your concern about realistic serving conditions and will try to revise the exposition to make the adaptation procedure, evaluation scope, and practical implications easier to follow.
The current experiments evaluate synchronous, greedy decoding, with supervision construction, training, weight publication, Reset restoration, and applicable outer-training costs included in end-to-end throughput. Section 4.3 already examines five domain-switching orders, including a four-domain stream. We agree, however, that sequential domain blocks do not fully capture interleaved heterogeneous traffic. We will rerun experiments with interleaved mixed-domain requests, using fixed adaptation settings and identical request orders.
Higher concurrency, batched inference, and stochastic sampling are valuable extensions. They require additional implementation and correctness validation, particularly for request-specific learner states, concurrent weight publication, and stochastic verification. Our priority in this revision is to establish robustness within the implemented execution setting and clarify the resulting claims. The paper’s central contribution is a restricted adaptation mechanism that uses verified-prefix supervision during generation, preserves reusable computation, and explicitly controls whether learned state is retained across requests. We will try to make this scope more prominent and discuss how the mechanism could be integrated into concurrent serving systems.
We sincerely thank you again for your time, thoughtful review, and constructive suggestions, which will help us strengthen the paper.