- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AdaSpec, which continually trains the drafter inside the inference engine. It reuses serving-path activations and target verification results, and reduces learning costs through gradient accumulation, fused kernels, and, in some configurations, overlapping learning with decoding. The algorithm directly optimizes expected acceptance length and uses online cost–benefit estimates to pause or resume training. Across two target–drafter pairs and seven workload groups, the authors report average acceptance-length gains of 10.6% and 14.7%, and average end-to-end throughput gains of 4.3% and 4.7%, including learning costs. The gains are largest on low-resource languages, while some general workloads and user streams do not improve in throughput.
Critical strengths
-
The paper turns online adaptation into a concrete inference-engine design and explicitly accounts for learning costs. Reusing verification supervision, choosing the trainable parameter scope, amortizing per-weight work, and maintaining consistency during weight updates all involve substantive technical work. End-to-end throughput including learning and the final drain is more persuasive than acceptance-length improvements alone.
-
The cost-aware continual learning mechanism addresses a real deployment trade-off. The method recognizes that continued training can cost more than it yields, and uses the gate, trainable-scope ablations, and traffic-shift experiments to examine when learning is worthwhile. The larger improvements on low-resource languages also support the motivation that greater drafter–traffic mismatch creates more room for adaptation.
Critical weaknesses
-
The frozen baseline does not fully use the GPU memory that would be available to it, leaving the practical resource advantage unestablished. Appendix A explicitly states that the KV-cache pool is reduced identically for AdaSpec and Frozen to accommodate optimizer state. This is a reasonable controlled comparison for isolating the computational cost of learning, but a deployed Frozen baseline does not need optimizer state and could use that memory for a larger KV cache and higher concurrency. With average throughput gains of only approximately 4%–5%, the absence of a full-memory-budget Frozen baseline could materially affect the conclusion.
-
Net throughput gains are not universal, and the reliability of the cost-aware gate needs stronger validation. Tables 7–9 in the appendix show that acceptance length can increase while throughput decreases for some users and tasks; examples include −0.5% on 27B WildChat and −2.8% on the 4B multi-developer stream. The average whole-stream gains are influenced by the larger benefits on low-resource languages. Evidence remains limited for rapidly interleaved or recurring traffic shifts, the break-even horizon on short streams, and run-to-run variability. Consequently, becoming faster the longer the server operates should not be interpreted as a general guarantee.
Questions for authors
-
Key baseline: What are the throughput–latency curves under long contexts and high concurrency when Frozen uses all available GPU memory, while AdaSpec bears the actual costs of optimizer state, activations, and training buffers? Please also report peak GPU memory usage.
-
Practical benefit: For each workload, what are the net break-even token count, confidence intervals across independent runs, fraction of time spent training under the gate, and probability of a throughput regression? Under rapidly interleaved traffic and recurring A→B→A shifts, can the gate limit losses and recover previously acquired benefits?
-
Objective clarification: How does the product of per-position distribution overlaps in Equations (1) and (7) correspond to the exact sequential acceptance probability when the target distribution depends on the sampled prefix? With the finite top-16 candidate set and comparisons of current and reference weights on the same drafts, are the quantities exact expectations or surrogate estimates? This affects the interpretation of both the training objective and the gate's signal.
4 comments
(Q3) Objective clarification: How does the product of per-position distribution overlaps in Equations (1) and (7) correspond to the exact sequential acceptance probability when the target distribution depends on the sampled prefix? With the finite top-16 candidate set and comparisons of current and reference weights on the same drafts, are the quantities exact expectations or surrogate estimates? This affects the interpretation of both the training objective and the gate's signal.
Thank you for this careful question. In short, Eq. 1 is exact for a greedy drafter and a surrogate for a sampling drafter; the top-16 sum is exact; and the gate's paired signal is a surrogate. Each choice is deliberate and validated empirically.
Prefix dependence (Eq. 1). Eq. 1 is the standard expression of Leviathan et al. (2023). We compute each overlap with on the actual drafted prefix, so the prefix dependence of is included. The only approximation is across positions: the exact probability multiplies the acceptance probabilities of the drafted tokens, , and averages over drafts, while Eq. 1 multiplies their means . For a greedy drafter (DFlash on Qwen3.5-4B), , so Eq. 1 is exact. For DFlash2, we use on purpose, because is a ratio of probabilities and makes the estimate and its gradient much noisier.
Top-16 candidates. DFlash2 samples only among its 16 candidates, so the sum over them is exact.
Why Eq. 7 is still the right objective. Eq. 7 directly rewards the overlap that sampled verification accepts, and it weights early positions more, since later ones count only after earlier ones are accepted. Cross-entropy and KL do neither. In Figure 7(c), Eq. 7 is the only objective that raises the realized acceptance length (+1.6%, versus −0.5% to −4.2%), and its estimate correlates with the realized acceptance length at 0.90 (Qwen3.8-27B) and 0.94 (Qwen3.5-4B).
Gate signal. The gate scores the current and reference weights on the same drafts, so per-draft difficulty cancels and the signal is much less noisy. Since the drafts come from the current weights, the signal is a surrogate, but the gate only compares it with a threshold.
We will state these points in §3 and add the derivation to the appendix.
We hope these answers address your concerns, and we are happy to clarify further.
Thanks for your rebuttal. I would keep my ratings.
Thank you for reading our rebuttal and for your reply! In response to your review, we added peak memory measurements, results on interleaved and A→B→A traffic, break-even and variance statistics, and a clarification of the objective. If any concerns remain, we would appreciate knowing which ones, so that we can address them during the discussion period.
Thank you for the careful review. We appreciate that you found the in-engine design technically substantive, the end-to-end accounting of learning persuasive, and the gate a response to a real deployment trade-off. We answer your questions as follows.
Clarifying the memory budget: at the concurrency levels we evaluate, the extra memory would not make Frozen faster. Its optimizer state and gradient buffers take 3.15 GB per GPU on Qwen3.8-27B and 3.2 GB on Qwen3.5-4B, about 4% of an 80 GB GPU. Measured on a mixed stream at C=8, the peak GPU memory on Qwen3.5-4B is 66.3 GB for Frozen and 69.5 GB for AdaSpec. In our evaluation across all tasks, no request is preempted in any run, and the prefix-cache hit rates (~92%) are identical across all methods. A Frozen server with a larger pool would therefore run at the same throughput, and the throughput–latency curves in Figure 5, where AdaSpec improves on Frozen at every concurrency up to C=64, already represent the full-memory Frozen baseline.
Frozen would gain from the memory only at a concurrency that just a full-pool server could fit, the same trade-off that speculative decoding makes for the drafter's own memory. We will add this discussion and the memory numbers to the final version.
We agree that the gain is not a guarantee, and we will qualify this claim. The gate is an online estimate, not an oracle: AdaSpec gains most on specialized, persistent traffic and can lose a few percent where the released drafter already fits it. We will revise "can become faster the longer it serves" in the abstract and conclusion to state this condition. The gain is not limited to low-resource languages, however: agent traffic also gains substantially, for example 21.6% in throughput on SWE-Gym and 8.1% on OSWorld with Qwen3.5-4B (Tables 8 and 9). To your specific questions (we will add the per-workload numbers to the appendix):
When translation returns, its acceptance length is 28.0% above Frozen in the first 100 requests, compared with 24.5% at the end of the first visit, so the earlier gain is recovered immediately rather than relearned.