- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpSpec reuses target KV instead of reconstructing a drafter's prefix KV from target hidden states. It adapts a five-layer JetSpec drafter with a root anchor and query adapters, and uses Operator Readout Distillation (ORD) to supervise token predictions, prefix readouts, attention normalizers, and query directions. On Qwen3-8B, it reports 3.5% higher accepted length and 2.9% higher decoding speed over JetSpec-N at tree budget 256. At 32K context, extra drafter memory drops from 0.91 to 0.032 GiB/request, allowing batch size to increase from 10 to 12 and throughput to improve by 9.7% at each method's maximum batch.
Critical strengths
- Reusing existing target KV removes redundant prefix draft kv. The memory reduction is concrete and can help when GPU memory limits batch size. Reusing target KV is also friendly to the hardware and infrastructure implementation.
- Readout and normalizer distillation are interesting and directly supervise how the drafter reads the cache. The evidence-switching experiments and cache interventions are useful attempts to examine the mechanism.
- The adaptation is lightweight. OpSpec reuses a pretrained JetSpec checkpoint and freezes both the target and the drafter backbone, training only the KV-interface adapters, root-anchor path, and normalizations.
Critical weaknesses
- Limited novelty. KVShot already makes essentially the same argument and motivation: hidden-state conditioning can obscure information needed by future draft steps, while re-attention over target KV shifts the difficulty to query estimation.
- Modest gains, with unclear attribution. Native KV with forward KL reaches τ=8.41 versus 8.34 for JetSpec-N; full ORD reaches 8.63. Most of the improvement appears after adding intermediate supervision and on-policy training, but comparable supervision for the baseline is not established. Individual additions improve τ by only about 0.35–1.18%, and the final speed gain is only 2.9%.
- Tree-only evaluation. There is no chain evaluation or small-budget serving comparison showing that the gain survives this cost under high load. Besides, the evaluation is limited to Qwen3-8B. More evaluation on latest models such as Qwen 3.8 27B which has large-scale RL training is preferred.
- Throughput gains mix memory capacity with execution efficiency. The largest gain, 9.7%, compares batch 12 against batch 10. At equal offered concurrency, gains are 3.0% and 4.8%; this still does not control active batch size. Saving drafter state is useful, but need not improve throughput once compute or communication becomes the bottleneck. The paper provides no evidence for distributed-KV or large-EP deployments, and does not sufficiently specify the timing setup or report absolute throughput.
Questions for authors
See weaknesses above.
2 comments
Response to Reviewer p1ud (2/2): Chain/Small-Budget Serving, 30B MoE Evaluation, and Active-Batch Controls
W3: Chain Evaluation, Small-Budget Serving Under Load, and 30B MoE Target Evaluation
Following precedents in Sequoia (Chen et al., NeurIPS 2024) and EAGLE-3 (Li et al., NeurIPS 2025), which show that chain and small-tree budgets are favored under concurrent serving load, we added two major evaluations:
- Chain and Small-Budget Serving at Confirmed Active Batch (Revised Table 22): We benchmarked a 16-token single chain alongside tree budgets of , , and nodes at a fixed, measured active batch size of on H100 GPUs.
| Proposal Structure (Measured Active Batch ) | OpSpec | (tok/s) | OpSpec (tok/s) | OpSpec Gain | OpSpec / AR | |
|---|---|---|---|---|---|---|
| Chain, 16 candidate tokens | 4.82 | 4.88 | 558.4 | 572.4 | +2.5% | 2.76x |
| Tree, 16 candidate nodes | 5.84 | 5.87 | 635.7 | 654.8 | +3.0% | 3.15x |
| Tree, 64 candidate nodes | 6.32 | 6.45 | 663.8 | 693.0 | +4.4% | 3.34x |
| Tree, 256 candidate nodes | 8.34 | 8.63 | 590.8 | 620.3 | +5.0% | 2.99x |
Consistent with Sequoia's throughput-versus-acceptance analysis, achieves peak serving throughput ( tok/s, over AR) at measured active batch , whereas maximizes . OpSpec outperforms in every topology, raising throughput by for 16-token chains, at tree , at tree , and at tree , confirming that gains hold for both single chains and small trees under concurrent load.
- Scaling to Qwen3-30B-A3B (Revised Table 12): Evaluating OpSpec on Qwen3-30B-A3B (30B MoE) across all seven benchmarks and 32K serving raises seven-task geometric mean from to (), lifts 32K throughput from to tok/s (), cuts incremental drafter state from to GiB/req (), and lowers total peak GPU memory from to GiB.
W4: Strictly Controlled Active-Batch Throughput, Absolute Metrics, and Deployment Scope
To separate execution speed from memory-enabled batch capacity (MagicDec, Sadhukhan et al., ICLR 2025; TETRIS, Wu et al., ACL 2025), revised Table 19 compares AR, , and OpSpec at identical measured active batches ( and at 2K, and at 32K), and revised Table 20 pairs matched active batch against each speculative method's own maximum feasible batch ( vs. ) on H100 GPUs with paged KV and a shared verifier.
- Matched Active Batch (No Capacity Mixing): At 2K context, OpSpec raises absolute throughput from tok/s () at active batch and tok/s () at active batch . At 32K context, throughput rises from tok/s () at active batch and tok/s () at matched active batch .
- Round Latency and Total Peak HBM: At 32K active batch , verification time is ms/round for both methods, while OpSpec lowers proposal time from to ms/round () and cuts measured total GPU peak memory (including model and target KV) from to GiB ( GiB, or of JetSpec-N's peak). Admitting active batch within GiB peak yields tok/s ().
- Deployment Scope: Our serving timings cover H100 paged-KV execution rather than distributed-KV disaggregation or multi-node expert-parallel deployments, where target KV access and cross-rank collectives remain governed by system topology; within HBM-resident serving, removing duplicate sequence-linear drafter caches ( GiB/req on 8B and GiB/req on 30B-A3B) directly improves both matched-batch throughput and feasible concurrency. We thank you for your constructive critique.
Response to Reviewer p1ud (1/2): Novelty Beyond KVShot and Symmetric Attribution Controls
We thank Reviewer p1ud for recognizing our concrete memory savings, hardware-friendly KV reuse, lightweight frozen-backbone adaptation, and readout/normalizer distillation with cache interventions. Below we address all four points with controlled factorials, chain and small-budget benchmarks, 30B MoE transfer, and matched active-batch measurements.
W1: Distinct Theoretical and Algorithmic Contributions Beyond KVShot
KVShot (Liu et al., COLM 2026 Workshop on Efficient Reasoning) compares hidden-only, cross-attended KV, and gated hybrid conditioning in an autoregressive EAGLE-3 setup and identifies draft query estimation as a bottleneck. OpSpec contributes distinct analytical, diagnostic, and supervisory advances:
W2: Symmetric Baseline Supervision and Clean Attribution of Gains
We agree that comparable intermediate and on-policy supervision must be tested on the reconstructed-KV baseline for symmetric attribution (DistillSpec, Zhou et al., ICLR 2024; EAGLE, Li et al., ICML 2024; HASS, Zhang et al., ICLR 2025; EAGLE-3, Li et al., NeurIPS 2025; Draft-OPD, Lei et al., EMNLP 2026). We ran two 2×2 factorials under the shared 1,000-step budget:
{FKL, Full ORD}with{Teacher-forced, On-policy final 180 steps}shows that at fixed state source, ORD adds +0.18 tokens teacher-forced (8.41→8.59, NRMSE 0.219→0.144) and +0.19 tokens on-policy (8.44→8.63, NRMSE 0.211→0.132), whereas on-policy branches alone add +0.03 and +0.04 tokens. Across all seven benchmarks (revised Table 11), point estimates improve everywhere, reaching +5.1% on MATH-500 (9.66×→10.09×), +4.7% on AIME 2025 (8.80×→9.16×), and +4.0% on LiveCodeBench (7.69×→7.95×).