acceptodds
Under review as a conference paper at ICLR 2027

AdaSpec: Speculative Decoding with In-Engine Continual Learning

Abstract

Speculative decoding accelerates large language model inference with a small drafter. However, the drafter is frozen after offline training, while it serves narrow and shifting traffic on which its predictions degrade. Recent work trains the drafter online outside the inference engine, which costs extra GPUs and data movement and serves with stale weights. We introduce AdaSpec, an inference engine in which the drafter learns continually from the traffic it serves, on the same GPUs and from the same forward passes. AdaSpec makes learning a by-product of serving: it reuses the activations and verification results already on the GPU and overlaps the backward pass with decoding. The drafter learns to maximize the expected number of accepted tokens, and a cost-aware gate trains it only while the gain pays for the cost. On streams that mimic real deployments, from agents to low-resource languages and real users, AdaSpec improves the acceptance length by 10.6% and 14.7% on average at two model scales, and the end-to-end throughput by 4.3% and 4.7%, including all learning costs. These results indicate that an inference server can become faster the longer it serves.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Continue Learning, Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QDyBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaSpec, which continually trains the drafter inside the inference engine. It reuses serving-path activations and target verification results, and reduces learning costs through gradient accumulation, fused kernels, and, in some configurations, overlapping learning with decoding. The algorithm directly optimizes expected acceptance length and uses online cost–benefit estimates to pause or resume training. Across two target–drafter pairs and seven workload groups, the authors report average acceptance-length gains of 10.6% and 14.7%, and average end-to-end throughput gains of 4.3% and 4.7%, including learning costs. The gains are largest on low-resource languages, while some general workloads and user streams do not improve in throughput.

Critical strengths
  1. The paper turns online adaptation into a concrete inference-engine design and explicitly accounts for learning costs. Reusing verification supervision, choosing the trainable parameter scope, amortizing per-weight work, and maintaining consistency during weight updates all involve substantive technical work. End-to-end throughput including learning and the final drain is more persuasive than acceptance-length improvements alone.

  2. The cost-aware continual learning mechanism addresses a real deployment trade-off. The method recognizes that continued training can cost more than it yields, and uses the gate, trainable-scope ablations, and traffic-shift experiments to examine when learning is worthwhile. The larger improvements on low-resource languages also support the motivation that greater drafter–traffic mismatch creates more room for adaptation.

Critical weaknesses
  1. The frozen baseline does not fully use the GPU memory that would be available to it, leaving the practical resource advantage unestablished. Appendix A explicitly states that the KV-cache pool is reduced identically for AdaSpec and Frozen to accommodate optimizer state. This is a reasonable controlled comparison for isolating the computational cost of learning, but a deployed Frozen baseline does not need optimizer state and could use that memory for a larger KV cache and higher concurrency. With average throughput gains of only approximately 4%–5%, the absence of a full-memory-budget Frozen baseline could materially affect the conclusion.

  2. Net throughput gains are not universal, and the reliability of the cost-aware gate needs stronger validation. Tables 7–9 in the appendix show that acceptance length can increase while throughput decreases for some users and tasks; examples include −0.5% on 27B WildChat and −2.8% on the 4B multi-developer stream. The average whole-stream gains are influenced by the larger benefits on low-resource languages. Evidence remains limited for rapidly interleaved or recurring traffic shifts, the break-even horizon on short streams, and run-to-run variability. Consequently, becoming faster the longer the server operates should not be interpreted as a general guarantee.

Questions for authors
  1. Key baseline: What are the throughput–latency curves under long contexts and high concurrency when Frozen uses all available GPU memory, while AdaSpec bears the actual costs of optimizer state, activations, and training buffers? Please also report peak GPU memory usage.

  2. Practical benefit: For each workload, what are the net break-even token count, confidence intervals across independent runs, fraction of time spent training under the gate, and probability of a throughput regression? Under rapidly interleaved traffic and recurring A→B→A shifts, can the gate limit losses and recover previously acquired benefits?

  3. Objective clarification: How does the product of per-position distribution overlaps in Equations (1) and (7) correspond to the exact sequential acceptance probability when the target distribution depends on the sampled prefix? With the finite top-16 candidate set and comparisons of current and reference weights on the same drafts, are the quantities exact expectations or surrogate estimates? This affects the interpretation of both the training objective and the gate's signal.

4 comments

Thank you for the careful review. We appreciate that you found the in-engine design technically substantive, the end-to-end accounting of learning persuasive, and the gate a response to a real deployment trade-off. We answer your questions as follows.


The frozen baseline does not fully use the GPU memory that would be available to it ... a deployed Frozen baseline does not need optimizer state and could use that memory for a larger KV cache and higher concurrency. With average throughput gains of only approximately 4%–5%, the absence of a full-memory-budget Frozen baseline could materially affect the conclusion. (Q1) What are the throughput–latency curves under long contexts and high concurrency when Frozen uses all available GPU memory, while AdaSpec bears the actual costs of optimizer state, activations, and training buffers? Please also report peak GPU memory usage.

Clarifying the memory budget: at the concurrency levels we evaluate, the extra memory would not make Frozen faster. Its optimizer state and gradient buffers take 3.15 GB per GPU on Qwen3.8-27B and 3.2 GB on Qwen3.5-4B, about 4% of an 80 GB GPU. Measured on a mixed stream at C=8, the peak GPU memory on Qwen3.5-4B is 66.3 GB for Frozen and 69.5 GB for AdaSpec. In our evaluation across all tasks, no request is preempted in any run, and the prefix-cache hit rates (~92%) are identical across all methods. A Frozen server with a larger pool would therefore run at the same throughput, and the throughput–latency curves in Figure 5, where AdaSpec improves on Frozen at every concurrency up to C=64, already represent the full-memory Frozen baseline.

Frozen would gain from the memory only at a concurrency that just a full-pool server could fit, the same trade-off that speculative decoding makes for the drafter's own memory. We will add this discussion and the memory numbers to the final version.


Net throughput gains are not universal, and the reliability of the cost-aware gate needs stronger validation. ... Evidence remains limited for rapidly interleaved or recurring traffic shifts, the break-even horizon on short streams, and run-to-run variability. Consequently, becoming faster the longer the server operates should not be interpreted as a general guarantee. (Q2) For each workload, what are the net break-even token count, confidence intervals across independent runs, fraction of time spent training under the gate, and probability of a throughput regression? Under rapidly interleaved traffic and recurring A→B→A shifts, can the gate limit losses and recover previously acquired benefits?

We agree that the gain is not a guarantee, and we will qualify this claim. The gate is an online estimate, not an oracle: AdaSpec gains most on specialized, persistent traffic and can lose a few percent where the released drafter already fits it. We will revise "can become faster the longer it serves" in the abstract and conclusion to state this condition. The gain is not limited to low-resource languages, however: agent traffic also gains substantially, for example 21.6% in throughput on SWE-Gym and 8.1% on OSWorld with Qwen3.5-4B (Tables 8 and 9). To your specific questions (we will add the per-workload numbers to the appendix):

  • Break-even. Where AdaSpec gains throughput, it overtakes Frozen after a median of 0.66M (Qwen3.8-27B) and 0.26M (Qwen3.5-4B) generated tokens, and after 0.19M and 0.15M on low-resource languages.
  • Fraction of time training. The gate trains in a median of 29% (Qwen3.8-27B) and 36% (Qwen3.5-4B) of the time.
  • Throughput regressions. In Tables 8 and 9, throughput decreases on 4 of 18 tasks on Qwen3.8-27B and 3 of 20 on Qwen3.5-4B, by at most 2.8%, and per user by at most 3.3% (Table 7). Acceptance increases in every case.
  • Confidence intervals. The 95% confidence interval of the acceptance gain excludes zero on all but one stream of each model; we will add repeated runs in the final version.
  • Rapidly interleaved traffic. On a stream that interleaves translation, news writing, and puzzles request by request, AdaSpec raises the acceptance length of every task and the throughput by 4.4%.
  • Recurring A→B→A shifts. We add a stream that switches from translation to news writing and back to translation (Qwen3.5-4B, 3,100 requests). AdaSpec raises the throughput in every segment, so the gate limits losses throughout: | Segment | Requests | Δτ | ΔThroughput | |---|---|---|---| | Translation, first visit | 1,200 | +14.3% | +0.4% | | News writing | 700 | +9.9% | +5.3% | | Translation, second visit | 1,200 | +34.3% | +13.9% | | Whole stream | 3,100 | +16.5% | +6.5% |

When translation returns, its acceptance length is 28.0% above Frozen in the first 100 requests, compared with 24.5% at the end of the first visit, so the earlier gain is recovered immediately rather than relearned.


(Q3) Objective clarification: How does the product of per-position distribution overlaps in Equations (1) and (7) correspond to the exact sequential acceptance probability when the target distribution depends on the sampled prefix? With the finite top-16 candidate set and comparisons of current and reference weights on the same drafts, are the quantities exact expectations or surrogate estimates? This affects the interpretation of both the training objective and the gate's signal.

Thank you for this careful question. In short, Eq. 1 is exact for a greedy drafter and a surrogate for a sampling drafter; the top-16 sum is exact; and the gate's paired signal is a surrogate. Each choice is deliberate and validated empirically.

Prefix dependence (Eq. 1). Eq. 1 is the standard expression of Leviathan et al. (2023). We compute each overlap αj\alpha_j with pjp_j on the actual drafted prefix, so the prefix dependence of pjp_j is included. The only approximation is across positions: the exact probability multiplies the acceptance probabilities of the drafted tokens, βi=min⁡(1,pi(di)/qi(di))\beta_i = \min(1, p_i(d_i)/q_i(d_i)), and averages over drafts, while Eq. 1 multiplies their means αi\alpha_i. For a greedy drafter (DFlash on Qwen3.5-4B), βi=αi\beta_i = \alpha_i, so Eq. 1 is exact. For DFlash2, we use αi\alpha_i on purpose, because βi\beta_i is a ratio of probabilities and makes the estimate and its gradient much noisier.

Top-16 candidates. DFlash2 samples only among its 16 candidates, so the sum over them is exact.

Why Eq. 7 is still the right objective. Eq. 7 directly rewards the overlap that sampled verification accepts, and it weights early positions more, since later ones count only after earlier ones are accepted. Cross-entropy and KL do neither. In Figure 7(c), Eq. 7 is the only objective that raises the realized acceptance length (+1.6%, versus −0.5% to −4.2%), and its estimate correlates with the realized acceptance length at 0.90 (Qwen3.8-27B) and 0.94 (Qwen3.5-4B).

Gate signal. The gate scores the current and reference weights on the same drafts, so per-draft difficulty cancels and the signal is much less noisy. Since the drafts come from the current weights, the signal is a surrogate, but the gate only compares it with a threshold.

We will state these points in §3 and add the derivation to the appendix.


We hope these answers address your concerns, and we are happy to clarify further.

Thanks for your rebuttal. I would keep my ratings.

Thank you for reading our rebuttal and for your reply! In response to your review, we added peak memory measurements, results on interleaved and A→B→A traffic, break-even and variance statistics, and a clarification of the objective. If any concerns remain, we would appreciate knowing which ones, so that we can address them during the discussion period.

Reviewer_uWFjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents AdaSpec, an inference engine that performs continual learning of the speculative decoding drafter directly inside the serving engine. The central observation is that speculative decoding already computes the drafter activations and target-model verification distributions, so these intermediate results can be reused for online training rather than sending data to a separate trainer. AdaSpec amortizes optimizer/weight-gradient costs over many drafted positions and overlaps the backward pass with subsequent decoding. It further introduces a cost-aware gate that decides when training is worthwhile based on the estimated improvement in acceptance length relative to its serving overhead.

Critical strengths

The cost-aware training mechanism is well motivated. The gate explicitly estimates whether the expected future benefit of adaptation exceeds the cost of training and can pause training after the drafter has adapted. The evaluation extensively covers multiple tasks such as agents, low-resource languages, coding, etc., making the effectivenss convincable. The paper compares different trainable parameter scopes and different learning objectives. In particular, the results suggest that training more parameters increases acceptance gain but eventually hurts throughput, supporting the need for a compute-aware choice of training scope.

Critical weaknesses
  1. The paper positions its main novelty as in-engine continual learning, whereas previous methods already exploit serving traffic for drafter adaptation. I am not fully convinced that the contribution is substantial.
  2. The evaluation was primarily upon a single task for each run. It is therefore unclear whether the reported gains persist under realistic heterogeneous workloads, especially when the task distribution changes over time.
  3. The paper mainly reports end-to-end throughput, which combines speculation benefits with the overhead of continual learning. A more detailed latency breakdown would make the results easier to interpret.
Questions for authors
  1. Does AdaSpec remain effective when serving multiple heterogeneous tasks/users simultaneously, rather than a single task per run?
  2. Could the authors provide a detailed latency/throughput breakdown separating drafting, target-model verification, continual-learning overhead, and other serving overheads?
  3. What is the unique contribution of AdaSpec compared with existing online/continual draft adaptation methods?
4 comments

Thank you for the review. We appreciate that you found the cost-aware gate well motivated, the evaluation across agents, low-resource languages, and coding convincing, and the comparison of trainable scopes and objectives useful. We answer your questions as follows.


The paper positions its main novelty as in-engine continual learning, whereas previous methods already exploit serving traffic for drafter adaptation. I am not fully convinced that the contribution is substantial. (Q3) What is the unique contribution of AdaSpec compared with existing online/continual draft adaptation methods?

Clarifying the contribution: AdaSpec is the first to show that drafter learning pays for itself on the same hardware. We agree that learning the drafter from served traffic is not new, and we credit this line of work in §2 (e.g., Online Speculative Decoding, Aurora, AdaFlash). These methods train the drafter outside the serving engine, on extra GPUs or in a separate distillation step, and report gains in acceptance length; the cost of learning is not part of their comparison. AdaSpec learns inside the serving engine, and every result includes the cost of learning. It also learns as it serves: the learner back-propagates the drafted positions right after each verification step, reusing what serving has already computed. Decoupled methods instead buffer the traffic, recompute the drafter's forward pass on a separate trainer, and periodically swap the new weights back, so the served drafter lags behind the traffic. This difference matters in practice: on the same two GPUs, the decoupled design (Aurora-DFlash) loses 36–46% of throughput, because its trainer GPU trains instead of serving, while AdaSpec gains up to 15% (Figure 6).

Learning inside the engine also raises problems that do not exist when training is free, and solving them is the technical contribution of the paper: how to reduce the cost of learning (from 51% to 2.8% of throughput, Figure 2), which parameters to train (Figure 7(a, b)), and when learning is worth its cost (the gate, §3.3). We will make this distinction explicit in the introduction.


The evaluation was primarily upon a single task for each run. It is therefore unclear whether the reported gains persist under realistic heterogeneous workloads, especially when the task distribution changes over time. (Q1) Does AdaSpec remain effective when serving multiple heterogeneous tasks/users simultaneously, rather than a single task per run?

Clarifying the target setting and the heterogeneity already in the evaluation. AdaSpec targets specialized traffic, such as one agent, the users of one product, or one language, because this is where a released drafter falls short and learning has the most room. Within this setting, our evaluation already includes heterogeneous traffic. The WildChat and SWE-chat streams replay the real chronological traffic of users whose conversations span many topics (Table 7), and the multi-developer stream serves several developers at the same time, raising acceptance by 11.6% (Table 9). For changes over time, the drafter re-adapts after each switch of language (Figure 8(b)), and on a new stream whose task changes from translation to news writing to puzzles, AdaSpec raises the throughput by 5.2% (see our response to Reviewer jiR1).

To test several tasks served at the same time, we also add a stream that randomly interleaves translation, news writing, and puzzles request by request, so that every batch mixes tasks (Qwen3.5-4B, 3,000 requests). The acceptance gain of each task stays close to its gain when the task is served alone on the same number of requests:

TaskRequestsΔτ, interleavedΔτ, served alone
Translation1,200+14.1%+12.2%
News writing700+16.2%+19.3%
Puzzles1,100+3.2%+6.9%
Average+11.2%+12.8%

Over the whole stream, AdaSpec raises the throughput by 4.4%, and by 8.0% on its second half. We will add this stream to the final version.


The paper mainly reports end-to-end throughput, which combines speculation benefits with the overhead of continual learning. A more detailed latency breakdown would make the results easier to interpret. (Q2) Could the authors provide a detailed latency/throughput breakdown separating drafting, target-model verification, continual-learning overhead, and other serving overheads?

We add the breakdown below. We report end-to-end throughput as the primary metric because it is what a deployment sees, and it already accounts for both the benefit of speculation and the cost of learning. To separate the two, we time each stage of every decoding step on the stream that switches from translation to news writing and back (Qwen3.5-4B, C=8; see our response to Reviewer QDyB). The table reports the GPU time per decoding step, averaged over all steps of the stream; the gate trains in 40% of them.

DraftingVerificationLearningOtherTotalTokens per step (τ)Time per token
Frozen1.24 ms5.40 ms–0.83 ms7.47 ms1.495.01 ms
AdaSpec1.25 ms5.42 ms0.37 ms0.89 ms7.93 ms1.714.64 ms

Verification dominates each step and is unchanged. Learning adds 0.37 ms per step on average, while each step yields 15% more tokens, so the time per generated token falls by 7%. We will add this breakdown to the final version.


We hope these answers address your concerns, and we are happy to clarify further.

Thank you for the response, which clarifies the multi-task scenario, time breakdown and contributions.

I am not fully convinced that the contribution is solid: I believe that drafters targeting specialized traffic should be trained offline with historical stats rather than online, and online training will definitely decrease initial speed and make TPOT unstable. Therefore, the central limitation is inherent to me.

I appreciate the authors' careful rebuttal, but I will maintain my score.

Thank you for the reply and for engaging with our rebuttal. We would like to briefly clarify the two points.

Offline vs. online training. We agree that offline training is needed, and AdaSpec starts from the offline-trained drafter. Offline training covers the majority of traffic, but specialized traffic is long-tailed and differs across deployments (an agent, a product, a language, a user), so a drafter cannot be trained offline for each of them in advance. AdaSpec complements offline training by adapting each engine to its own traffic, without a separate data or training pipeline.

Speed and stability. Online learning costs some speed compared with no learning, but adaptation should be compared at equal hardware: offline or decoupled training needs extra GPUs that could otherwise serve requests. On the same two GPUs, the decoupled design loses 36–46% of throughput while AdaSpec gains up to 15% (Figure 6), because AdaSpec reuses the activations and verification results that serving already computes.The initial cost is short: AdaSpec overtakes Frozen after a median of 0.26M–0.66M generated tokens. The cost of learning is also bounded, at about 3% of throughput on average, and zero while the gate pauses.

We see this trade-off as the central problem that in-engine continual learning raises, and the paper addresses it through the co-design of the system and the learning algorithm. Thank you again for the careful review.

Reviewer_jiR1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that frozen drafters lose acceptance on narrow, shifting deployment traffic, and that online methods such as Aurora need separate training GPUs. AdaSpec trains the drafter on the serving GPUs. It reuses what serving has already computed, accumulates gradients over many drafted positions before each optimizer step, and overlaps the backward pass with decoding, cutting the cost of learning from 51% to 2.8% of throughput (Figure 2). The learner maximizes the expected acceptance length (Eq. 7), and a gate trains only while the measured gain exceeds a break-even threshold. On two model pairs, the authors report average gains of 10.6% and 14.7% in acceptance length and 4.3% and 4.7% in throughput.

Critical strengths
  • The system design makes in-engine learning practical. Amortizing the per-weight work alone cuts the cost from 51% to 9.1% (Figure 2), and Appendix A handles deployment issues such as tensor-parallel consistency. This supports the main premise that the drafter can learn on the serving GPUs cheaply.

  • The cost accounting is honest. All learning time is in the wall-clock measurement, both arms serve the same requests at the same time, and negative cases are reported per task and per user (Tables 7 to 9). This makes the numbers credible.

Critical weaknesses
  • The throughput gain is small outside low-resource languages, especially on Qwen3.8-27B, and the gate is not tested under fast traffic shifts. On Qwen3.8-27B, the six other groups average about 1.1% in throughput (3.4% on Qwen3.5-4B), 5 of the 9 real users have a negative throughput change (Table 7), and no variance is reported. With margins this small the gate matters, but the only shift experiment is one translation stream with two switches and no throughput (Figure 8(b)), and gate versus always training is shown on two streams only (Figure 7). The gate's prior probe interval is about 1.1M tokens on Qwen3.8-27B, over half of each 2M-token stream. I would like to see a fast-shifting stream that compares the gate, always training, a fixed schedule, and the frozen drafter over several runs.

  • The evidence for the new objective is thin. Eq. 7 is used only on Qwen3.8-27B, and its only ablation (Figure 7(c)) shows a 1.6% acceptance gain on an unspecified stream, far below the 10.6% average in Table 1. Comparing it with cross-entropy on several Table 1 streams would help.

Questions for authors
  • How much is the KV-cache pool reduced for the optimizer state (Appendix A), and is the frozen drafter faster at C=64 with the full pool?

  • The Qwen3.5-4B learner does not overlap the backward pass with decoding (Appendix A). What is its learning cost in the format of Figure 2?

2 comments

Thank you for the detailed and constructive review. We appreciate that you found that the system design makes in-engine learning practical, and that the cost accounting is honest, with all learning time included and negative cases reported per task and per user. We answer your questions as follows.

The throughput gain is small outside low-resource languages, especially on Qwen3.8-27B, and the gate is not tested under fast traffic shifts. On Qwen3.8-27B, the six other groups average about 1.1% in throughput (3.4% on Qwen3.5-4B), 5 of the 9 real users have a negative throughput change (Table 7), and no variance is reported.

Thank you for this careful reading; we will clarify the following points in the final version.

The whole-stream gain includes the cost of learning from the first request. Each stream ends after 2M generated tokens, so its first part is spent learning before the adapted drafter pays it back (Figure 4). The gain is larger on the second half of each stream, after the drafter has adapted (Table 10).

The gain depends on how far the traffic is from the drafter's training data, not only on the language. Low-resource languages are the extreme case, but agent traffic also gains substantially: SWE-Gym (OpenHands) gains 21.6% in throughput on Qwen3.5-4B and 4.8% on Qwen3.8-27B, and OSWorld gains 8.1% on Qwen3.5-4B (Tables 8 and 9). After adaptation, AdaSpec serves 13% more tokens per second on OSWorld at the same per-user speed (Figure 5). Open-ended chat has little shared structure, so the drafter has less to learn within 2M tokens. We will extend the streams beyond 2M tokens to study whether such harder traffic can be learned with more tokens.

Real users. Acceptance rises for all nine users on both models (Table 7). On Qwen3.8-27B, the five users with a throughput loss lose between 0.7% and 2.9%, because the gate pauses training once the gain no longer pays for its cost.

Variance. The 95% confidence interval of the acceptance gain excludes zero on all but one stream of each model. We will report confidence intervals and repeated runs in the final version.


With margins this small the gate matters, but the only shift experiment is one translation stream with two switches and no throughput (Figure 8(b)), and gate versus always training is shown on two streams only (Figure 7). The gate's prior probe interval is about 1.1M tokens on Qwen3.8-27B, over half of each 2M-token stream. I would like to see a fast-shifting stream that compares the gate, always training, a fixed schedule, and the frozen drafter over several runs.

Clarifying how the gate reacts to a shift. The 1.1M-token interval belongs to the periodic probe, which only guards against slow drift that no single window reveals. An abrupt shift is detected separately: a CUSUM test compares each window's acceptance with its running mean and restarts training as soon as the level departs from it (Algorithm 1, lines 15 to 17). The gate therefore does not wait for the next periodic probe. We will make this distinction explicit in §3.3.

New results on shifting traffic. Following your suggestion, we add a stream whose task changes twice, from translation to news writing to puzzles (Qwen3.5-4B). The gate detects each switch within two windows (about 10 seconds) and restarts training, and AdaSpec raises the throughput in every segment:

SegmentRequestsΔτΔThroughput
Translation2,400+23.0%+6.1%
News writing700+20.1%+10.4%
Puzzles1,400+3.1%+1.2%
Whole stream4,500+12.3%+5.2%

We also compare the gate with always training on three more streams. Always training reaches a higher acceptance length but loses throughput once the gain saturates, whereas the gate keeps the throughput gain (gain over Frozen, throughput / acceptance length):

StreamGateAlways
APIGen-MT+0.9% / +8.8%+0.3% / +14.3%
Reasoning Gym+4.1% / +8.0%−3.1% / +10.6%
WildChat user+2.0% / +5.8%−5.0% / +7.7%

We will add these results, together with a fixed-schedule baseline and repeated runs, to the final version.


The evidence for the new objective is thin. Eq. 7 is used only on Qwen3.8-27B, and its only ablation (Figure 7(c)) shows a 1.6% acceptance gain on an unspecified stream, far below the 10.6% average in Table 1. Comparing it with cross-entropy on several Table 1 streams would help.

Clarifying why Eq. 7 is used only on Qwen3.8-27B. Eq. 7 maximizes the acceptance of a drafter that samples its drafts, which depends on how much the drafter's distribution overlaps with the target's. DFlash2 (Qwen3.8-27B) samples among its top-16 candidates, so this overlap is what matters. DFlash (Qwen3.5-4B) drafts greedily: a drafted token is accepted with the target's probability of that token, which depends only on the drafter's top token, and cross-entropy toward the target's top token already trains for it.

Clarifying Figure 7(c). It is evaluated on single-turn math data, following DFlash. The released drafter is already well trained on such data, so there is little room to adapt, and Figure 7(c) mainly verifies that the objective behaves as intended: Eq. 7 raises the acceptance length, whereas cross-entropy and the other objectives lower it. The 10.6% in Table 1, in contrast, averages over workloads where the drafter fits the traffic much worse. We will state this in the caption and add more ablations of the objective on Table 1 streams in the final version.


How much is the KV-cache pool reduced for the optimizer state (Appendix A), and is the frozen drafter faster at C=64 with the full pool?

No, the full pool would not make Frozen faster at C=64. The optimizer state and gradient buffers take 3.2 GB on Qwen3.5-4B and 3.15 GB per GPU on Qwen3.8-27B. Even at C=64, no request is preempted, and the prefix-cache hit rate is the same for both methods (~91.2%), so a larger pool would leave the throughput of Frozen unchanged. We will add these numbers to Appendix A.


The Qwen3.5-4B learner does not overlap the backward pass with decoding (Appendix A). What is its learning cost in the format of Figure 2?

Measured as in Figure 2, the average learning cost on Qwen3.5-4B is about 7% of throughput when the learner always trains, and about 3% with the gate. We will add the details to Figure 2 in the final version.


We hope these answers address your concerns, and we are happy to clarify further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.