acceptodds
Desk rejected at ICLR 2027

TraceDraft: Online Draft Adaptation During Speculative Decoding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Speculative decoding, which verifies draft tokens in a single target-model pass, has been widely adopted as an effective method for accelerating language model inference. Online distillation has been used to adapt speculative drafts to the requests encountered during deployment. However, methods that train on completed requests or request batches defer the benefit of their feedback to subsequent requests. Consequently, exploiting token-level feedback before the current response ends, which enables immediate adaptation from ongoing inference without first generating complete responses for training, remains a challenge. As real request streams often span multiple domains, retaining useful adaptation across domain changes is also worth addressing. To tackle these challenges, we propose TraceDraft, a lightweight framework for online draft adaptation within responses. TraceDraft uses target predictions on the prompt and verified continuation as supervision, updating only the draft's output normalization and prediction head. Two training modes are available to deal with different situations: Reset mode confines adaptation to each conversation, while Persistent mode retains updates across requests with a built-in anchor toward the initial weights. Updates of TraceDraft remain separate from outer training, enabling combination with other online learners. We evaluate eight workloads, including AIME2024/2025 and MT-Bench. Experimental results show gains in accepted length and end-to-end throughput on all eight workloads, with long-output gains on DeepSeek-R1-Distill-Llama-8B averaging 10.84% and 9.16%, respectively, over the fixed draft. Combining TraceDraft with online learners also adds up to 10.03% throughput over their respective baselines. In cross-domain experiments, Persistent improves throughput by up to 21.11% over the full stream and up to 12.10% after the domain switch relative to the fixed draft. These results suggest that TraceDraft provides a practical framework for learning from incoming requests to improve speculative decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Online Knowledge Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qnCDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents TraceDraft, a lightweight online adaptive framework for speculative decoding. TraceDraft adapts the draft model mid-generation via token-level supervision from the target model, without waiting for fully completed responses. To reduce training overhead while retaining KV cache reuse, the framework only updates the draft model’s output normalization layer and prediction head. It supports Reset and Persistent modes for single-conversation and cross-domain multi-request scenarios. Experiments demonstrate that TraceDraft improves accepted sequence length and end-to-end throughput across eight benchmarks on DeepSeek-R1-Distill-Llama-8B and Vicuna-13B-v1.3.

Critical strengths
  1. The adaptation mechanism is lightweight. TraceDraft only updates the output normalization layer and prediction head while keeping the draft backbone frozen, preserving existing KV caches. The updated parameters are immediately applied to the remaining response.
Critical weaknesses
  1. This paper conducts experiments only under greedy sampling with batch size = 1 and synchronous updates. It does not validate the performance under high-concurrency, batched inference or stochastic sampling settings.
Questions for authors
  1. Can the authors provide end-to-end results under more realistic serving conditions, including interleaved mixed-domain request streams, higher concurrency, larger batch sizes, and stochastic sampling? Such experiments would help assess whether the reported gains generalize beyond the current evaluation settings.
5 comments

Thank you for taking time to review our paper and for recognizing the benefit of restricting adaptation to the output layers. We appreciate your concern about realistic serving conditions and will try to revise the exposition to make the adaptation procedure, evaluation scope, and practical implications easier to follow.

The current experiments evaluate synchronous, greedy decoding, with supervision construction, training, weight publication, Reset restoration, and applicable outer-training costs included in end-to-end throughput. Section 4.3 already examines five domain-switching orders, including a four-domain stream. We agree, however, that sequential domain blocks do not fully capture interleaved heterogeneous traffic. We will rerun experiments with interleaved mixed-domain requests, using fixed adaptation settings and identical request orders.

Higher concurrency, batched inference, and stochastic sampling are valuable extensions. They require additional implementation and correctness validation, particularly for request-specific learner states, concurrent weight publication, and stochastic verification. Our priority in this revision is to establish robustness within the implemented execution setting and clarify the resulting claims. The paper’s central contribution is a restricted adaptation mechanism that uses verified-prefix supervision during generation, preserves reusable computation, and explicitly controls whether learned state is retained across requests. We will try to make this scope more prominent and discuss how the mechanism could be integrated into concurrent serving systems.

We sincerely thank you again for your time, thoughtful review, and constructive suggestions, which will help us strengthen the paper.

AuthorsOpenReview ↗

Response to Q1 and W1

Dear reviewer, thank you for your constructive suggestion to evaluate TraceDraft under more realistic serving conditions. Following your advice, we have added experiments with interleaved mixed-domain requests, larger batch size, increased concurrency, and stochastic sampling.

For each serving configuration, we construct a mixed-domain stream of 120 questions sampled from LongBench-Write, AIME-2024, AIME-2025, PubMedQA, and TheoremQA, and shuffle their order. The subsets and ordering vary across configurations, while each baseline–Persistent pair processes exactly the same requests in the same order.

We evaluate B=2, C=2 and B=2, C=4, where B is the maximum verification batch size and C is the maximum number of in-flight conversations. Both configurations are tested with greedy decoding (temperature=0, top-p=1) and stochastic sampling (temperature=0.7, top-p=0.9). We use DeepSeek-R1-Distill-Llama-8B with its matched EAGLE-3 draft. Other settings remain fixed. Persistent uses a learning rate of 10⁻⁶, anchor coefficient 0.01, and one adaptation step after five verification rounds in each eligible turn.

Table 1 reports accepted length and domain-specific throughput speedup relative to the matched fixed EAGLE-3 baseline.

Table 1. Accepted length and throughput speedup under batched, concurrent generation (greedy and sampling).

Setting (B=2)DatasetBaseline avg. lengthPersistent avg. lengthSpeedup
C=2, GreedyLongBench-Write1.261.471.18×
C=2, GreedyAIME-20242.012.181.05×
C=2, GreedyAIME-20252.122.211.02×
C=2, GreedyPubMedQA3.673.841.02×
C=2, GreedyTheoremQA2.322.471.06×
C=2, SamplingLongBench-Write1.531.711.15×
C=2, SamplingAIME-20241.821.921.04×
C=2, SamplingAIME-20251.872.061.09×
C=2, SamplingPubMedQA3.573.791.07×
C=2, SamplingTheoremQA2.172.281.03×
C=4, GreedyLongBench-Write1.311.591.23×
C=4, GreedyAIME-20242.102.231.09×
C=4, GreedyAIME-20251.982.131.07×
C=4, GreedyPubMedQA3.673.911.09×
C=4, GreedyTheoremQA2.092.311.08×
C=4, SamplingLongBench-Write1.391.501.10×
C=4, SamplingAIME-20241.821.891.06×
C=4, SamplingAIME-20251.761.831.02×
C=4, SamplingPubMedQA3.603.701.03×
C=4, SamplingTheoremQA2.152.251.07×

As is shown in the results above, TraceDraft improves accepted length and throughput in all 20 workload–configuration comparisons, with speedups ranging from 1.02× to 1.23×. These gains are consistent in direction and broadly comparable in magnitude to those reported in the paper. In particular, positive gains persist under stochastic sampling and when requests from different domains are interleaved.

We also record the actual verification batch sizes and in-flight conversation counts to verify the execution conditions. Table 2 reports these measurements.

Table 2. Observed batch size and concurrency.

Setting (B=2)MethodMean actual batchMax actual batchMean in-flightPeak in-flight
C=2, GreedyBaseline1.96921.9692
C=2, GreedyPersistent1.97721.9772
C=2, SamplingBaseline1.97021.9702
C=2, SamplingPersistent1.98421.9842
C=4, GreedyBaseline1.95323.8534
C=4, GreedyPersistent1.98123.9284
C=4, SamplingBaseline1.98923.9454
C=4, SamplingPersistent1.99023.9594

As is demonstrated, the observed verification batches remain close to two requests, and the mean in-flight counts reach 3.853–3.959 when C=4. These measurements confirm sustained batched execution with multiple concurrent conversations.

Together, these experiments provide evidence that TraceDraft’s benefits extend beyond the original single-request, greedy evaluation. The shared adaptation configuration retains gains under the tested mixed-domain, batched, concurrent, and stochastic settings. We sincerely thank you again for suggesting this evaluation, which strengthens the evidence for TraceDraft’s practical applicability.

Thank you for the additional experiments, which partially address my concern. However, all new experiments use a verification batch size of B=2. My concern about scaling to larger batches therefore remains. Experiments with larger verification batches would provide more convincing evidence. I maintain my current score.

Reviewer_WFQ1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes TraceDraft for online draft adaptation during speculative decoding. It uses target supervision on the prompt and verified continuation to update the draft's output layers while preserving decoding caches, with configurable state reuse and integration with outer online learners.

Critical strengths
  1. The design integrates online distillation into speculative decoding, allowing feedback from an ongoing response to improve its remaining generation while preserving decoding caches.
  2. Experiments across eight workloads and two target models show throughput gains after accounting for adaptation overhead, and demonstrate complementary gains with existing online learners.
Critical weaknesses
  1. The evaluation provides limited evidence of robustness to adaptation hyperparameters. Learning rates vary across workloads and modes, while stability under continued updates is not characterized. This leaves uncertainty about reliable throughput gains without workload-specific tuning.
Questions for authors
  1. How were the adaptation hyperparameters selected? Can a shared configuration retain the gains across workloads and domain switches?
  2. What evidence shows that Persistent updates remain stable over successive requests, particularly when domains change? How often are updates rolled back?
5 comments

Thank you for your careful and constructive assessment. We particularly appreciate your recognition of both the within-response adaptation mechanism and the complementary throughput gains after adaptation overhead is included. Your questions about configuration robustness and stability identify an important opportunity to make the empirical evidence more useful for deployment.

We agree that the parameter-selection procedure and sensitivity results should be clearer. Several settings are already shared: Appendix A specifies a learning rate of 10−610^{-6} for the short-workload main comparisons and Persistent ablations, alongside common distillation and update-control settings. We also conducted six-learning-rate sweeps for Reset and Persistent. We will try to consolidate these results, explain the search and selection procedure, and distinguish shared settings from workload-specific choices.

Regarding stability, Persistent includes anchoring to the initial output weights, gradient clipping, and rollback checks for invalid or excessive updates. Table 2 already evaluates state reuse across several domain transitions, although aggregate throughput does not fully characterize the learning trajectory. We will supplement these results with rollback frequencies and normalized parameter drift over successive requests.

We sincerely thank you again for your time, thoughtful review, and constructive suggestions, which will help us strengthen the paper.

AuthorsOpenReview ↗

Response to Q1: Hyperparameter selection and a shared configuration

Dear reviewer, we have supplemented our evaluation with a mixed-domain experiment to address the questions. Thank you for your careful review and for encouraging us to examine shared configurations and the stability of continued adaptation more directly.

As noted in our earlier response, we selected the adaptation hyperparameters through learning-rate sweeps and extensive experiments with different update frequencies and freezing strategies. Across the tested settings, the best-performing configurations for different workloads tended to fall within similar hyperparameter ranges. These observations guided the final settings reported in the paper and motivated our evaluation of a shared configuration.

Following your suggestion, we constructed a new stream of 120 questions, randomly sampled from AIME-2024, AIME-2025, PubMedQA, TheoremQA, and LongBench-Write, and shuffled them into an interleaved request sequence. Both methods processed the same questions in the same order using DeepSeek-R1-Distill-Llama-8B and greedy decoding.

We used Persistent mode with a learning rate of 10−610^{-6}, anchor coefficient (λ=0.01\lambda=0.01), and update interval (f=1) throughout the stream. Each turn permits at most one optimizer step, triggered after five verification rounds. These settings remained fixed across domains.

DatasetBaseline avg. lengthPersistent avg. lengthThroughput speedup
LongBench-Write1.371.461.05×
AIME-20241.942.151.10×
AIME-20252.092.201.05×
PubMedQA3.663.831.07×
TheoremQA2.012.431.15×

Speedups are measured against the fixed EAGLE-3 baseline on the same mixed stream and include online adaptation overhead. Each row summarizes requests from the corresponding domain within that stream.

As is shown in the results, TraceDraft improves accepted length and throughput on all five domains. These gains are comparable in scale to those reported in the paper, showing that a shared configuration can preserve useful adaptation benefits on the tested mixed-domain stream without domain-specific tuning.

AuthorsOpenReview ↗

Response to Q2: Persistent stability and rollback frequency

We examined update stability on the same 120-question mixed stream using the shared configuration above. Persistent retained its adapted weights and optimizer state across requests, including changes in domain. We measured the maximum relative displacement of the trainable parameters from their initial values throughout the stream.

Stability metricPersistent, f=1
Maximum relative drift of output normalization weights0.000004
Maximum relative drift of prediction-head weights0.001907

As is demonstrated, the maximum relative displacement was 0.0004% for the output normalization weights and 0.1907% for the prediction head. These small displacements, together with the positive gains across domains, support stable continued adaptation under the shared configuration. Persistent remained stable over this mixed-domain run, and no update triggered rollback.

Persistent uses anchoring to the initial weights, gradient clipping, and rollback checks for nonfinite or excessively large updates. These checks remain active for every attempted update, even when the observed updates are stable and no rollback is required.

We sincerely thank you again for your constructive suggestions, which helped us make the evidence for configuration robustness and adaptation stability more explicit.

Reviewer_4UotOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the problem of online draft adaptation in speculative decoding by proposing TraceDraft, a lightweight framework for dynamically updating the draft model. The work presents two key innovations: first, it selectively updates only the RMS normalization layers and the prediction head of the draft model; second, it introduces a state lifecycle paradigm and multiple fine-grained update timing mechanisms, thereby further enhancing the acceleration gains of speculative decoding. Experiments across 16 configurations validate the effectiveness of the proposed approach, making a meaningful contribution to online adaptation techniques within the field of speculative decoding.

Critical strengths

This work demonstrates that more efficient Test-Time Speculation can be achieved by leveraging verified prefixes as supervision signals while exclusively updating the prediction head.

Critical weaknesses

The contribution of the core design regarding test-time update strategies appears limited.

First, the work does not provide an automated strategy for selecting update timings. In the experiments, each benchmark is effectively treated as a single-domain scenario, but real scenes may not follow single domain. Without this assumption, the absence of an automatic selection mechanism would prevent the framework from achieving the speedups reported in the experiments.

Second, the concept of test-time update timing is not novel to this paper.

Thus, this work is more appropriately positioned as an extended analytical study of prior Test-Time Speculation research.

Questions for authors
  1. The results in Table 2 indicate that the relative advantages of different update strategies vary across scenarios. How should the system determine which strategy to employ during actual deployment?

  2. How can the authors convince the research community that the TraceDraft approach is superior to the method proposed in the TTS paper?

  3. I noticed that certain deductions, such as Propositions 1 and 2, are formatted as standalone theoretical proofs. Do the authors feel this level of formalization is strictly necessary?

5 comments

Thank you for the time and care you have devoted to reviewing our work. We appreciate your recognition of the value of lightweight adaptation from verified prefixes. Your questions about update selection, the relationship to TTS, and the presentation of our theoretical results are helpful for strengthening both the practical contribution and the clarity of the paper.

  1. Update timing and deployment

Thank you for this helpful suggestion. We agree that finding out how the system should choose an appropriate schedule when the request distribution changes would make our work more solid. We have conducted extensive experiments on learning rates, update frequencies, and freezing strategies, and will now try to use these results to investigate a mechanism. Across several benchmarks, we have observed improvements at similar learning-rate scales, suggesting that a shared configuration may provide a useful starting point without requiring separate tuning for every workload. We will examine this consistency more carefully and use the frequency and freezing results to identify when continued updates are worthwhile.

  1. Relationship to TTS and the advantages of TraceDraft

Thank you for your kind advice on TTS. We consider that our advantages lie in how adaptation is performed, retained, and combined with other learning processes. We will try to provide a detailed analysis.

  1. Presentation of Propositions 1 and 2

We appreciate your suggestion regarding the level of formalization. Propositions 1 and 2 specify the conditions under which decoding caches remain valid and the outer training trajectory remains unchanged. Their purpose is to make the correctness requirements of these mechanisms explicit. We agree that these straightforward arguments can be presented more concisely. We will try to shorten their treatment.

We sincerely thank you again for your time, thoughtful review, and constructive suggestions, which will help us strengthen the paper.

AuthorsOpenReview ↗

Response to Q1 and W1

Dear reviewer, thank you for highlighting the practical question of how to select an update strategy when incoming requests span different domains. Following your suggestion, we conducted additional experiments to test whether a fixed, shared configuration can retain the gains reported in the paper on interleaved mixed-domain requests.Our earlier learning-rate sweeps and extensive scheduling experiments found that effective configurations across workloads generally fell within similar hyperparameter ranges, which motivated testing a shared configuration without adjusting it for individual domains.

We constructed three separately sampled and shuffled mixed-domain streams, each containing 120 questions, to simulate heterogeneous incoming requests. Mix 1 includes AIME-2024, AIME-2025, PubMedQA, TheoremQA, and LongBench-Write. Mix 2 and Mix 3 contain newly sampled questions from the first four datasets. These are different question samples.

We evaluated two fixed schedules: P-every1, which permits an update on every turn, and P-every8, which permits an update every eight turns. Both use Persistent mode, a learning rate of 10−610^{-6}, anchor coefficient 0.01, and at most one optimizer step on each selected turn, triggered after five verification rounds. All other adaptation settings remain fixed across streams and domains. No domain labels are used to select the update schedule.

All experiments use DeepSeek-R1-Distill-Llama-8B and greedy decoding. Within each stream, all methods process the same questions in the same input order.

Mix 1: 120 questions from five datasets

DatasetBaseline avg. lengthP-every1 avg. lengthP-every1 speedupP-every8 avg. lengthP-every8 speedup
LongBench-Write1.371.461.05×1.391.04×
AIME-20241.942.151.10×2.121.08×
AIME-20252.092.201.05×2.341.08×
PubMedQA3.663.831.07×3.841.08×
TheoremQA2.012.431.15×2.321.11×

Mix 2: 120 newly sampled questions from four datasets

DatasetBaseline avg. lengthP-every1 avg. lengthP-every1 speedupP-every8 avg. lengthP-every8 speedup
AIME-20241.942.071.06×1.981.03×
AIME-20252.042.301.09×2.161.04×
PubMedQA3.703.831.04×3.841.04×
TheoremQA2.142.381.08×2.321.07×

Mix 3: 120 newly sampled questions from four datasets

DatasetBaseline avg. lengthP-every1 avg. lengthP-every1 speedupP-every8 avg. lengthP-every8 speedup
AIME-20242.032.411.13×2.111.03×
AIME-20252.182.271.04×2.271.03×
PubMedQA3.763.971.06×3.981.06×
TheoremQA2.412.641.07×2.621.05×

Both fixed schedules improve accepted length and throughput for every reported domain in all three streams. P-every1 achieves throughput gains of 4% to 15%, comparable in scale to those reported in the paper. Thus, useful speedups persist when questions from different domains are interleaved and the same configuration is applied throughout.

P-every1 also provides the stronger overall result: the equal-domain macro-average throughput gains are 8.40%, 6.75%, and 7.50% across the three streams, compared with 7.80%, 4.50%, and 4.25% for P-every8. These averages summarize the per-domain relative gains. P-every1 has higher reported throughput in nine of the thirteen stream–domain comparisons, ties in two, and trails in two. This supports using P-every1 as a shared deployment default, without selecting a different schedule for each incoming domain.

We further evaluated the same fixed P-every1 configuration at {B, C}={4, 4} and {8, 8}, where B denotes the verification batch size and C denotes the maximum number of concurrent requests. We retained greedy decoding and the same adaptation hyperparameters throughout. TraceDraft improved accepted length and achieved throughput gains of 4–10% over matched baselines in all ten dataset–setting comparisons, including adaptation overhead. These results show that the shared configuration remains effective under larger-batch and concurrent execution, without retuning it for individual domains or serving configurations. Detailed results and actual batch and concurrency statistics are provided in our response to Reviewer 3.

These results address the concern that the reported gains require a single-domain workload or a favorable strategy selected separately for each benchmark. A fixed Persistent configuration retains comparable gains across the tested heterogeneous streams.

We sincerely thank you again for this suggestion, which helped us provide a more concrete answer to the deployment question.

AuthorsOpenReview ↗

Response to Q2 and W2

Thank you for encouraging us to clarify the relationship between TraceDraft and TTS. We appreciate TTS’s valuable diagnosis of draft degradation during long generation and its demonstration that online distillation can address this problem. TraceDraft makes a distinct methodological contribution through the joint design of restricted output-layer adaptation, reusable learning state, and isolation from outer online training.

First, TraceDraft provides a concrete design for adaptation under a limited optimization budget. It constructs aligned supervision from the prompt and committed continuation, balances these two sources, and updates only the output normalization and prediction head. The draft’s feature extractor remains fixed, so gradients do not pass through its attention or MLP backbone, and committed-prefix KV caches remain valid. Each selected turn permits at most one update, after which the adapted head immediately serves the remaining generation. This connects the choice of supervision and trainable parameters to explicit control over the computational work required for adaptation.

Second, TraceDraft makes learned state reusable across independent requests. TTS’s request-level design resets adaptation between independent requests to isolate their updates. TraceDraft additionally provides Persistent mode, which carries output weights and optimizer state across requests, with anchoring to the initial weights and update checks. This enables useful calibration to survive beyond the request that supplied its supervision. Our domain-switching experiments further examine the benefits of retaining this state as incoming requests change.

Third, TraceDraft introduces an explicit separation between serving adaptation and the parameter trajectory of an outer online learner. Temporary serving weights improve ongoing generation, while the outer learner continues training from its original parent checkpoint. Under the deterministic greedy conditions specified in our isolation proposition, this preserves the outer training trajectory. Our paired checks confirm identical generated tokens and outer checkpoints, while combinations with OSD and Ens-EAGLE-3 achieve additional end-to-end throughput gains of up to 10.03% over their respective baselines, including adaptation and outer-training costs. This establishes a practical benefit of combining learning within a response with learning across completed requests.

Together, these mechanisms and their empirical validation support TraceDraft’s distinct methodological contribution in how adaptation is computed, retained across requests, and integrated with outer online learners with demonstrated benefits for constrained update budgets and continuing request streams.

We appreciate TTS’s contribution to understanding draft degradation during long generation, and we welcome the authors’ release of their implementation as a valuable resource for the community. At the time of our submission, this official implementation was not available: the repository was created and linked from the authors’ project page on November 7, 2026, after our submission. Our manuscript already cites TTS, and we commit to updating the citation to its NeurIPS 2026 version and including a direct experimental comparison using the official implementation in the camera-ready paper. We will conduct this comparison under matched evaluation settings, accounting for adaptation overhead, to clarify the computational trade-offs and benefits of the two approaches. We regard TTS as an important contribution to adaptive speculative decoding and sincerely thank the reviewer for encouraging a closer comparison with this work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Experiments and Clarifications

We sincerely thank all reviewers (4Uot, WFQ1, and qnCD) for their careful assessments and constructive suggestions. Their feedback has helped us strengthen the empirical evidence for TraceDraft and clarify its methodological contribution. During the discussion, we conducted additional experiments addressing shared configurations, adaptation stability, and deployment beyond single-request greedy decoding.

To address configuration robustness and update-strategy selection (Reviewers 4Uot and WFQ1), we evaluated three independently sampled and shuffled mixed-domain streams, each containing 120 questions. We applied a fixed P-every1 configuration throughout: Persistent mode, a learning rate of 10−610^{-6}, anchor coefficient of 0.01, and at most one update per turn after five verification rounds. No domain labels or domain-specific configuration changes were used. This shared configuration improves accepted length and throughput in every reported domain, achieving throughput gains of 4%–15%, broadly comparable in scale to those reported in the paper. These results support its use as a practical default for heterogeneous request streams. Regarding adaptation stability (Reviewer WFQ1), the maximum normalized head drift ranged from 0.001867 to 0.002012 across the three streams, while the maximum normalization-layer drift remained 0.000004.

In response to concerns about larger batches, concurrency, and stochastic sampling (Reviewer qnCD), we conducted supplementary experiments under four configurations: (B,C)=(2,2),(2,4),(4,4),(8,8), where B denotes verification batch size and C denotes the concurrency limit. Each configuration evaluates both greedy decoding and stochastic sampling with temperature 0.7 and top-p 0.9, using identical questions and input orders within each baseline–Persistent comparison. Across all 40 domain–setting comparisons, TraceDraft improves accepted length and achieves throughput gains. Specifically, gains remain positive at 4%–12% for batch sizes 4 and 8. Recorded verification batches and in-flight session counts confirm that these experiments exercise batched and concurrent execution. These results support the effectiveness of the shared adaptation configuration across the tested serving conditions.

We also clarified TraceDraft’s relationship to TTS and the presentation of its theoretical arguments (Reviewer 4Uot). We recognize TTS’s valuable contribution to adaptation during generation. Our technical comparison emphasizes TraceDraft’s restricted output-layer adaptation, cross-request state reuse, and isolation from outer online training, supported by demonstrated complementary gains with existing online learners. At the time of our submission, the official TTS implementation was not yet publicly available. We have committed to citing its NeurIPS 2026 version and including a matched comparison using its official implementation in the camera-ready manuscript. For the theoretical presentation, we provided proposed replacement text that expresses Proposition 1 in two sentences and substantially shortens Proposition 2 while preserving its correctness assumptions.

We sincerely appreciate the reviewers’ time and thoughtful engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.