acceptodds
Under review as a conference paper at ICLR 2027

The Prefix Is Already Cached: Reusing Target KV for Speculative Decoding

Abstract

A compact drafter can make several proposals before verifying against a single target model. Head-based speculative drafters use the target hidden states to build their proposals, to reconstruct a verified-prefix representation. Target model inference already cached the prefix representation in its own native key-value (KV) states. In same-query experiments, we show that a system with duplicate hidden KV states can change how future tokens are interacting with earlier context, and more importantly, how these states can cause inaccuracies in verified prefix readouts when queried by the target itself. Particularly we find that when a continuation visits information receiving low attention at the current token location, readouts can be subject to larger inaccuracies in these hidden KV state reconstructions, causing non-negligible errors in rejecting later steps. Based on these observations, we introduce OpSpec, which removes the need for prefix reconstruction, where each branch simply queries the target's native KV states without any sequence linear drafter state over the verified token sequence. Operator Readout Distillation is used to distill the target model's prefix readouts with queries over these KV states, and a fused attention kernel allows for caching pages without copying. OpSpec extends accepted length at 7 benchmarks on Qwen3-8B at the reported tree budget. Much larger gains are observed on tasks that require a larger re-visiting of earlier context. At 32K context length, OpSpec saves 96.5% of incremental drafter state, allowing 20% more maximum batch size, and 9.7% more throughput at each method's maximum feasible batch.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, efficient LLM inference, KV cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p1udOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpSpec reuses target KV instead of reconstructing a drafter's prefix KV from target hidden states. It adapts a five-layer JetSpec drafter with a root anchor and query adapters, and uses Operator Readout Distillation (ORD) to supervise token predictions, prefix readouts, attention normalizers, and query directions. On Qwen3-8B, it reports 3.5% higher accepted length and 2.9% higher decoding speed over JetSpec-N at tree budget 256. At 32K context, extra drafter memory drops from 0.91 to 0.032 GiB/request, allowing batch size to increase from 10 to 12 and throughput to improve by 9.7% at each method's maximum batch.

Critical strengths
  • Reusing existing target KV removes redundant prefix draft kv. The memory reduction is concrete and can help when GPU memory limits batch size. Reusing target KV is also friendly to the hardware and infrastructure implementation.
  • Readout and normalizer distillation are interesting and directly supervise how the drafter reads the cache. The evidence-switching experiments and cache interventions are useful attempts to examine the mechanism.
  • The adaptation is lightweight. OpSpec reuses a pretrained JetSpec checkpoint and freezes both the target and the drafter backbone, training only the KV-interface adapters, root-anchor path, and normalizations.
Critical weaknesses
  • Limited novelty. KVShot already makes essentially the same argument and motivation: hidden-state conditioning can obscure information needed by future draft steps, while re-attention over target KV shifts the difficulty to query estimation.
  • Modest gains, with unclear attribution. Native KV with forward KL reaches τ=8.41 versus 8.34 for JetSpec-N; full ORD reaches 8.63. Most of the improvement appears after adding intermediate supervision and on-policy training, but comparable supervision for the baseline is not established. Individual additions improve τ by only about 0.35–1.18%, and the final speed gain is only 2.9%.
  • Tree-only evaluation. There is no chain evaluation or small-budget serving comparison showing that the gain survives this cost under high load. Besides, the evaluation is limited to Qwen3-8B. More evaluation on latest models such as Qwen 3.8 27B which has large-scale RL training is preferred.
  • Throughput gains mix memory capacity with execution efficiency. The largest gain, 9.7%, compares batch 12 against batch 10. At equal offered concurrency, gains are 3.0% and 4.8%; this still does not control active batch size. Saving drafter state is useful, but need not improve throughput once compute or communication becomes the bottleneck. The paper provides no evidence for distributed-KV or large-EP deployments, and does not sufficiently specify the timing setup or report absolute throughput.
Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer p1ud (1/2): Novelty Beyond KVShot and Symmetric Attribution Controls

We thank Reviewer p1ud for recognizing our concrete memory savings, hardware-friendly KV reuse, lightweight frozen-backbone adaptation, and readout/normalizer distillation with cache interventions. Below we address all four points with controlled factorials, chain and small-budget benchmarks, 30B MoE transfer, and matched active-batch measurements.

W1: Distinct Theoretical and Algorithmic Contributions Beyond KVShot

KVShot (Liu et al., COLM 2026 Workshop on Efficient Reasoning) compares hidden-only, cross-attended KV, and gated hybrid conditioning in an autoregressive EAGLE-3 setup and identifies draft query estimation as a bottleneck. OpSpec contributes distinct analytical, diagnostic, and supervisory advances:

  1. Subspace Non-Identifiability and Same-Query Isolation. KVShot compares end-to-end autoregressive conditioning modes where query estimation and prefix fidelity vary together. Section 2.2 and Appendix E.4–E.5 prove that at fixed keys, rank-rr observed prefix attention probability rows (r<tr<t) leave a (t−r)dv(t-r)d_v-dimensional value subspace AΔV=0A\Delta V=0 (plus orthogonal key directions in Appendix E.5 when keys vary), meaning any new attention row outside Row⁡(A)\operatorname{Row}(A) can have its readout altered by a perturbation annihilated on observed rows. Appendix D.1 (Table 6) and Table 1 test this mechanism empirically through same-target-query replay (0.00020.0002 native vs. 0.1860.186 reconstructed NRMSE), controlled evidence switching, and TopM future-specific cache masking (8.68.6 pp drop vs. 1.41.4 pp random mask).
  2. Operator Readout Distillation (ORD) Addresses Query Estimation Directly. Rather than leaving target-KV reads to output-token losses or hybrid caches, OpSpec aligns branch queries in frozen target WQW_Q/QNorm/RoPE coordinates, combines shared prefix pages and local speculative ancestors via the shared-denominator softmax log-sum-exp identity (FlashInfer, Ye et al., MLSys 2025), and supervises normalized prefix readouts r^P,vj\hat r_{P,v}^j, log normalizers log⁡Z^P,vj\log \hat Z_{P,v}^j, and query directions. On an independent target-KV drafter DKVD_{\mathrm{KV}} (revised Table 16), replacing matched token-level supervision with ORD cuts readout NRMSE from 0.2260.226 to 0.1510.151, raises τ\tau from 8.078.07 to 8.278.27, and lifts end-to-end throughput from 677.6677.6 to 694.8694.8 tok/s (+2.5%+2.5\%).
  3. Zero-Prefix-State Causal-Parallel Tree Drafting: Unlike sequential autoregressive drafting, OpSpec adapts a single-pass causal-parallel tree drafter (JetSpec, Hu et al., 2026) with zero sequence-linear prefix payload, yielding a +0.17+0.17-token interaction (ΔDiD\Delta_{\mathrm{DiD}}, p=0.0024p=0.0024, Table 4) when branch-specific queries diverge from depth-mean queries.

W2: Symmetric Baseline Supervision and Clean Attribution of Gains

We agree that comparable intermediate and on-policy supervision must be tested on the reconstructed-KV baseline for symmetric attribution (DistillSpec, Zhou et al., ICLR 2024; EAGLE, Li et al., ICML 2024; HASS, Zhang et al., ICLR 2025; EAGLE-3, Li et al., NeurIPS 2025; Draft-OPD, Lei et al., EMNLP 2026). We ran two 2×22\times2 factorials under the shared 1,000-step budget:

  • Symmetric ORD + On-Policy Supervision on JetSpec-N (Revised Table 3): Adding full ORD and the 180-step on-policy stage to reconstructed-KV JetSpec-N raises τ256\tau_{256} from 8.348.34 (7.33×7.33\times) to 8.498.49 (7.43×7.43\times). Under the identical full ORD + on-policy recipe, OpSpec reaches τ256=8.63\tau_{256}=8.63 (7.54×7.54\times), beating the ORD-trained baseline by +0.14+0.14 tokens and +0.11×+0.11\times speedup (twice the +0.07+0.07 gap under FKL alone). Within each prefix source, full ORD + on-policy adds +0.22+0.22 tokens on native KV versus +0.15+0.15 on reconstructed KV.
  • Isolating ORD from On-Policy Rollouts on Native KV (Revised Table 13): Crossing {FKL, Full ORD} with {Teacher-forced, On-policy final 180 steps} shows that at fixed state source, ORD adds +0.18+0.18 tokens teacher-forced (8.41→8.598.41\rightarrow8.59, NRMSE 0.219→0.1440.219\rightarrow0.144) and +0.19+0.19 tokens on-policy (8.44→8.638.44\rightarrow8.63, NRMSE 0.211→0.1320.211\rightarrow0.132), whereas on-policy branches alone add +0.03+0.03 and +0.04+0.04 tokens. Across all seven benchmarks (revised Table 11), point estimates improve everywhere, reaching +5.1%+5.1\% on MATH-500 (9.66×→10.09×9.66\times\rightarrow10.09\times), +4.7%+4.7\% on AIME 2025 (8.80×→9.16×8.80\times\rightarrow9.16\times), and +4.0%+4.0\% on LiveCodeBench (7.69×→7.95×7.69\times\rightarrow7.95\times).
AuthorsOpenReview ↗

Response to Reviewer p1ud (2/2): Chain/Small-Budget Serving, 30B MoE Evaluation, and Active-Batch Controls

W3: Chain Evaluation, Small-Budget Serving Under Load, and 30B MoE Target Evaluation

Following precedents in Sequoia (Chen et al., NeurIPS 2024) and EAGLE-3 (Li et al., NeurIPS 2025), which show that chain and small-tree budgets are favored under concurrent serving load, we added two major evaluations:

  1. Chain and Small-Budget Serving at Confirmed Active Batch b⋆=8b_\star=8 (Revised Table 22): We benchmarked a 16-token single chain alongside tree budgets of 1616, 6464, and 256256 nodes at a fixed, measured active batch size of b⋆=8b_\star=8 on H100 GPUs.
Proposal Structure (Measured Active Batch b⋆=8b_\star=8)J⋆J^\star τ\tauOpSpec τ\tauJ⋆J^\star (tok/s)OpSpec (tok/s)OpSpec GainOpSpec / AR
Chain, 16 candidate tokens4.824.88558.4572.4+2.5%2.76x
Tree, 16 candidate nodes5.845.87635.7654.8+3.0%3.15x
Tree, 64 candidate nodes6.326.45663.8693.0+4.4%3.34x
Tree, 256 candidate nodes8.348.63590.8620.3+5.0%2.99x

Consistent with Sequoia's throughput-versus-acceptance analysis, B=64B=64 achieves peak serving throughput (693.0693.0 tok/s, 3.34×3.34\times over AR) at measured active batch 88, whereas B=256B=256 maximizes τ\tau. OpSpec outperforms J⋆J^\star in every topology, raising throughput by +2.5%+2.5\% for 16-token chains, +3.0%+3.0\% at tree 1616, +4.4%+4.4\% at tree 6464, and +5.0%+5.0\% at tree 256256, confirming that gains hold for both single chains and small trees under concurrent load.

  1. Scaling to Qwen3-30B-A3B (Revised Table 12): Evaluating OpSpec on Qwen3-30B-A3B (30B MoE) across all seven benchmarks and 32K serving raises seven-task geometric mean τ256\tau_{256} from 8.128.12 to 8.458.45 (+4.1%+4.1\%), lifts 32K throughput from 431.8431.8 to 450.9450.9 tok/s (+4.4%+4.4\%), cuts incremental drafter state from 0.4620.462 to 0.0240.024 GiB/req (−94.8%-94.8\%), and lowers total peak GPU memory from 78.278.2 to 76.476.4 GiB.

W4: Strictly Controlled Active-Batch Throughput, Absolute Metrics, and Deployment Scope

To separate execution speed from memory-enabled batch capacity (MagicDec, Sadhukhan et al., ICLR 2025; TETRIS, Wu et al., ACL 2025), revised Table 19 compares AR, J⋆J^\star, and OpSpec at identical measured active batches (11 and 1616 at 2K, 11 and 1010 at 32K), and revised Table 20 pairs matched active batch 1010 against each speculative method's own maximum feasible batch (1010 vs. 1212) on H100 GPUs with paged KV and a shared verifier.

  • Matched Active Batch (No Capacity Mixing): At 2K context, OpSpec raises absolute throughput from 445.9→457.0445.9\rightarrow457.0 tok/s (+2.5%+2.5\%) at active batch 11 and 1007.8→1045.11007.8\rightarrow1045.1 tok/s (+3.7%+3.7\%) at active batch 1616. At 32K context, throughput rises from 258.6→266.4258.6\rightarrow266.4 tok/s (+3.0%+3.0\%) at active batch 11 and 690.4→723.5690.4\rightarrow723.5 tok/s (+4.8%+4.8\%) at matched active batch 1010.
  • Round Latency and Total Peak HBM: At 32K active batch 1010, verification time is 49.249.2 ms/round for both methods, while OpSpec lowers proposal time from 41.741.7 to 39.339.3 ms/round (−5.8%-5.8\%) and cuts measured total GPU peak memory (including model and target KV) from 78.678.6 to 69.869.8 GiB (−8.8-8.8 GiB, or 11.2%11.2\% of JetSpec-N's peak). Admitting active batch 1212 within 78.978.9 GiB peak yields 757.4757.4 tok/s (+9.7%+9.7\%).
  • Deployment Scope: Our serving timings cover H100 paged-KV execution rather than distributed-KV disaggregation or multi-node expert-parallel deployments, where target KV access and cross-rank collectives remain governed by system topology; within HBM-resident serving, removing duplicate sequence-linear drafter caches (0.910→0.0320.910\rightarrow0.032 GiB/req on 8B and 0.462→0.0240.462\rightarrow0.024 GiB/req on 30B-A3B) directly improves both matched-batch throughput and feasible concurrency. We thank you for your constructive critique.
Reviewer_9EkDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpSpec, which enables causal-parallel speculative drafters to directly reuse the target model's native KV cache instead of reconstructing a separate prefix representation. The method uses query adapters and Operator Readout Distillation (ORD) to improve draft-side prefix reads, together with a split-attention implementation that reduces incremental drafter memory. Experiments on Qwen3-8B with a JetSpec-based drafter demonstrate improved accepted length, reduced incremental memory, and modest throughput gains.

Critical strengths
  1. A well-motivated analysis of prefix reconstruction errors. The paper identifies a concrete limitation of hidden-state-based prefix reconstruction and supports it through same-query measurements, evidence-switching experiments, and controlled ablations. These analyses help explain why direct access to target KV can improve speculative proposal quality.

  2. Meaningful reduction in redundant prefix state. Removing the sequence-linear drafter cache is particularly relevant to memory-constrained inference. The reported increase in feasible batch size demonstrates that the optimization can provide practical capacity benefits beyond improvements in accepted length.

Critical weaknesses
  1. The novelty relative to existing target-KV-based drafting methods and the generality beyond JetSpec remain insufficiently established. Direct reuse of target KV is not new. GliDe with a CaPE (Du et al., 2024) already introduces cross-attention to the target's cached KV during speculative drafting. LongSpec (Yang et al., 2026) combines target-KV cross-attention with a bounded drafter cache and an attention aggregation mechanism that separates prefix and speculative-token computations. More closely related to the paper's motivation, KVShot (Liu et al., 2026) compares hidden-only, KV-only, and hybrid reuse and identifies long-range information loss and query estimation as important drafting bottlenecks. OpSpec contributes a branch-conditioned interface and readout distillation for causal-parallel drafting, but it is not sufficiently clear how much of the contribution is a generalizable mechanism beyond these existing approaches, rather than an architectural adaptation to JetSpec. The evaluation on only Qwen3-8B and a JetSpec-based drafter further limits the evidence for broader applicability.

  2. The incremental serving performance improvement is modest and insufficiently characterized. The improvement from 7.33× to 7.54× over AR corresponds to approximately 2.9% additional speedup over JetSpec-N. The larger 9.7% throughput gain compares each method at its own maximum feasible batch, while the 96.5% memory reduction refers specifically to incremental drafter state rather than total serving memory. These are valid but distinct improvements. Without absolute throughput, total memory consumption including target KV, and detailed serving configurations, it is difficult to assess how much practical serving benefit the method provides beyond reducing duplicated drafter state.

Questions for authors
  1. Could the authors clarify the specific methodological advances over GliDe with a CaPE, LongSpec, and KVShot? In particular, LongSpec already demonstrates target-KV reuse with bounded drafter memory and prefix/local attention aggregation, while KVShot investigates information loss under hidden-state conditioning and identifies query prediction as a major limitation. I acknowledge the ORD ablations in Table 3 and the controlled prefix-source experiments. Which additional findings or mechanisms are unique to OpSpec's causal-parallel formulation, and which conclusions should be expected to generalize to other target-KV-based drafters?

  2. Could the authors evaluate OpSpec on an additional target model or drafter configuration, reporting accepted length, end-to-end throughput, and incremental memory under matched conditions? This would help establish whether the improvement is specific to JetSpec or transferable to other speculative drafting architectures. If such transfer requires substantially different interfaces or training, clarifying these requirements would also be valuable.

  3. Could the authors provide absolute throughput and total peak GPU memory, including target KV, at both matched batch/concurrency and each method's maximum feasible batch? Please also specify the serving framework, GPU/TP configuration, batching policy, CUDA Graph settings, and proposal/verification latency breakdown. These measurements would help distinguish the benefits of faster execution from the additional batch capacity enabled by reduced drafter memory.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9EkD (1/2): Advances Over Prior Target-KV Methods and Cross-Architecture Transfer

We thank Reviewer 9EkD for highlighting our prefix-reconstruction analysis, evidence-switching and cache interventions, and the batch-capacity benefits of removing duplicate drafter state. Below we address each question with new cross-model, cross-drafter, and serving measurements.

W1 & Q1: Methodological Advances Over GliDe, LongSpec, and KVShot, and Generalizability

We appreciate the opportunity to clarify how OpSpec advances beyond prior target-KV drafters and which components generalize across architectures:

  1. Advances Over GliDe with a CaPE (Du et al., ICML 2024) and LongSpec (Yang et al., ACL 2026): GliDe inserts cross-attention to upper target KV alongside confidence-aware proposal expansion, and LongSpec combines target-KV cross-attention with bounded sliding-window local draft attention in autoregressive drafting. OpSpec maps branch states directly into frozen target attention spaces (preserving target WQW_Q, target QNorm, and stored RoPE coordinates) and combines shared native prefix pages with branch-local speculative ancestors through the shared-denominator softmax merge identity via log-sum-exp reduction (FlashInfer, Ye et al., MLSys 2025; DeFT, Yao et al., ICLR 2025). Crucially, OpSpec introduces Operator Readout Distillation (ORD) to supervise normalized prefix readouts r^P,vj\hat r_{P,v}^j, prefix log normalizers log⁡Z^P,vj\log \hat Z_{P,v}^j, and query cosine directions, directly calibrating the eλP−λe^{\lambda_P-\lambda} and eλLv−λe^{\lambda_{L_v}-\lambda} merge weights between verified prefix and speculative ancestors.
  2. Advances Over KVShot (Liu et al., COLM 2026 Workshop on Efficient Reasoning). KVShot compares hidden-only, cross-attended target-KV, and gated hybrid conditioning autoregressively and identifies draft query estimation as a primary bottleneck. OpSpec advances this line on three fronts. First, Section 2.2 and Appendix E.4–E.5 characterize prefix non-identifiability analytically by showing that at fixed keys, rank-rr observed prefix attention probability rows (r<tr<t) leave a (t−r)dv(t-r)d_v-dimensional value subspace AΔV=0A\Delta V=0 (and unobserved orthogonal key directions in Appendix E.5 when keys vary), such that for any new attention row outside Row⁡(A)\operatorname{Row}(A) there exists an unobserved perturbation that alters its readout. Second, Appendix D.1 (Table 6) and Table 1 pair empirical same-target-query measurements (0.1860.186 reconstructed vs. 0.00020.0002 native NRMSE) with controlled evidence-switching and TopM cache interventions. Third, ORD directly supervises branch queries against native prefix readouts and log normalizers, cutting readout NRMSE from 0.2190.219 to 0.1440.144 teacher-forced and from 0.2110.211 to 0.1320.132 on-policy on Qwen3-8B (Table 13), and from 0.2260.226 to 0.1510.151 on an independent target-KV drafter (Table 16).
  3. Causal-Parallel Specifics vs. General Target-KV Mechanisms. Single-pass ancestor masking, root-anchor initialization, and the cross-branch query interaction (ΔDiD=+0.17\Delta_{\mathrm{DiD}}=+0.17 tokens between branch-specific and depth-mean states on our switchable checkpoint, p=0.0024p=0.0024, Table 4) target parallel multi-branch tree drafters (JetSpec, Hu et al., 2026; cf. hidden-state reconstruction in block-diffusion DFlare, Zhang et al., EMNLP 2026). Meanwhile, frozen target WQW_Q/QNorm coordinate alignment with shared-denominator LSE merging, ORD supervision of prefix readouts and log normalizers, and in-place paged target-KV reuse transfer directly to general target-KV drafters.

W1 & Q2: Transfer to Qwen3-30B-A3B and to an Independent Non-JetSpec Drafter

Following your suggestion, we evaluated transferability on a second target model and an independent target-KV drafter under matched conditions:

  • Additional Target Model (Qwen3-30B-A3B, Revised Table 12): Evaluating OpSpec against a matched hidden-conditioned baseline J⋆J^\star on Qwen3-30B-A3B (30B MoE) across all seven benchmarks at B=256B=256 and in 32K serving raises seven-task geometric mean τ256\tau_{256} from 8.128.12 to 8.458.45 (+4.1%+4.1\%), increases 32K serving throughput from 431.8431.8 to 450.9450.9 tok/s (+4.4%+4.4\%), cuts incremental drafter state by 94.8%94.8\% (0.462→0.0240.462\rightarrow0.024 GiB/req), and reduces measured total peak GPU memory from 78.278.2 to 76.476.4 GiB.
  • Independent Target-KV Drafter (DKVD_{\mathrm{KV}}, Revised Table 16): Comparing ORD against matched token-level supervision on the same independent target-KV drafter DKVD_{\mathrm{KV}} (0.0570.057 GiB/req incremental state) lowers prefix-readout NRMSE from 0.2260.226 to 0.1510.151, raises τ\tau from 8.078.07 to 8.278.27, and increases end-to-end throughput from 677.6677.6 to 694.8694.8 tok/s (+2.5%+2.5\%).
AuthorsOpenReview ↗

Response to Reviewer 9EkD (2/2): Absolute Throughput, Total GPU Memory, and Latency Breakdown

W2 & Q3: Absolute Throughput, Total Peak GPU Memory, and Per-Round Latency Breakdown

We completely agree that reporting absolute tok/s, total HBM peak memory including target KV and weights, strictly matched active batches, and per-round proposal/verification latency makes the practical serving benefits transparent, in line with serving evaluation standards in Sequoia (Chen et al., NeurIPS 2024), MagicDec (Sadhukhan et al., ICLR 2025), TETRIS (Wu et al., ACL 2025), and EAGLE-3 (Li et al., NeurIPS 2025). We have added revised Tables 19 and 20 to report these exact measurements on H100 GPUs.

1. Absolute Throughput at Strictly Matched Active Batch Sizes (Revised Table 19, Qwen3-8B):

Context LengthActual Active BatchTarget AR (tok/s)JetSpec-N / J⋆J^\star (tok/s)OpSpec (tok/s)OpSpec Gain vs. J⋆J^\starOpSpec / AR
2K1128.4445.9457.0+2.5%3.56x
2K16291.61007.81045.1+3.7%3.58x
32K182.7258.6266.4+3.0%3.22x
32K10218.9690.4723.5+4.8%3.31x

2. Total Peak HBM Memory and Proposal/Verification Latency Breakdown at 32K Context (Revised Table 20):

Method & Operating Point (32K)Actual Active BatchOutput Throughput (tok/s)Total Peak GPU Mem (GiB)Incremental Drafter State (GiB/req)Proposal (ms/round)Verification (ms/round)
JetSpec-N (Matched & Max Batch)10690.478.60.91041.749.2
OpSpec (Matched Batch)10723.569.80.03239.349.2
OpSpec (Own Max Feasible Batch)12757.478.90.03243.657.7

These measurements cleanly separate execution speed from batch-capacity gains:

  • Execution Speed at Matched Active Batch 10: At 32K context and matched active batch 1010, verification time is 49.249.2 ms/round for both methods because both share the exact same paged-KV verifier, whereas OpSpec's fused split-attention kernel removes hidden fusion and contextual draft KV projection, lowering proposal latency from 41.741.7 to 39.339.3 ms/round (−5.8%-5.8\%). Combined with higher accepted length per round, output throughput rises from 690.4690.4 to 723.5723.5 tok/s (+4.8%+4.8\%).
  • Total Peak GPU Memory Including Model and Target KV: At 32K context and active batch 1010, JetSpec-N reaches 78.678.6 GiB measured total HBM peak versus 69.869.8 GiB for OpSpec, saving 8.88.8 GiB (11.2%11.2\% of JetSpec-N's measured total peak, matching 10×(0.910−0.032)=8.7810 \times (0.910 - 0.032) = 8.78 GiB of incremental drafter state).
  • Capacity-Enabled Throughput at Active Batch 12: Those 8.88.8 GiB of freed HBM allow OpSpec to run at active batch 1212 within a comparable 78.978.9 GiB total HBM peak (versus 78.678.6 GiB for JetSpec-N at batch 1010). Even with batch-12 round times of 43.643.6 ms proposal and 57.757.7 ms verification, total output throughput reaches 757.4757.4 tok/s (+9.7%+9.7\% over JetSpec-N's 690.4690.4 tok/s capacity limit).
  • Documented Serving Platform and Metadata. Our serving experiments run on H100 GPUs with paged KV caching, a shared verification engine across speculative baselines, and OpSpec's fused prefix/local split-attention kernel reading target KV pages in place without copying at the measured active batches in Tables 19 and 20 (11, 1010, 1212, 1616). For completeness regarding the remaining configuration fields you asked about, specific GPU count and HBM SKU, serving precision (BF16 is specified for training), framework version, tensor-parallel degree, batching scheduler, and CUDA Graph settings are not separately recorded in the manuscript.
Reviewer_FXYKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates differences between verified-prefix KV reconstructed from target hidden features and the target’s native KV cache, examining their relationship to branch-dependent prefix access and draft acceptance. The authors propose OPSPEC, a JetSpec-based causal-parallel drafter that reuses target KV through per-layer query adapters, alongside split attention and an Operator Readout Distillation objective. Experiments on Qwen3-8B report modest decoding speed improvements and substantial reductions in incremental drafter memory, although the contributions of native KV reuse and additional training components remain difficult to separate.

Critical strengths
  • The diagnostic analysis is well designed. Applying the same target query to native and reconstructed prefix caches isolates prefix-readout mismatch from draft-query prediction error.

  • The proposed method is practical and memory-efficient, as it reuses the target’s existing KV cache.

Critical weaknesses
  • The attribution of performance gains to native KV reuse remains unclear. In Table 3, OpSpec trained with forward KL alone achieves a 7.36× speedup, compared with 7.33× for JetSpec-N in Table 2—an improvement of approximately 0.4%. If these conditions use matched forward-KL objectives and training settings, this suggests that the architectural changes, including native KV reuse, provide only a modest speed benefit. The robustness of this small difference is difficult to assess. The larger speedup of 7.54× is obtained after adding further training components, leaving their contribution relative to native KV reuse insufficiently isolated.

  • The high-level idea of conditioning a drafter on the target’s existing KV cache is established in prior work. GliDe explicitly demonstrates acceptance improvements from target kv cache access, while LongSpec also motivates KV sharing as reducing contextual-representation discrepancies and additional memory. The paper should distinguish its specific reconstruction-error diagnosis and readout-supervision method from these existing ideas, and clarify the incremental architectural contribution.

Questions for authors
  • Same as W1 and W2

  • Table 4 is not explicitly referenced in the text, and its relationship to the named baselines is unclear. Can the authors specify the underlying model and explain how its conditions relate to DDTree, JetSpec-N, and OpSpec?

  • What exact objective is used to train JetSpec-N? Does it match Table 3’s forward-KL-only condition in temperature, node weighting, and training histories?

  • Can the authors provide baseline and OpSpec accepted lengths and speedups for all seven benchmarks at budget 256, including GSM8K and HumanEval? These results help assess the claimed task-level ranking and whether aggregate gains conceal regressions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FXYK (1/2): Attribution of Gains and Distinctions from Prior Target-KV Methods

We thank Reviewer FXYK for recognizing the strength of our same-query diagnostic design and the memory efficiency of reusing native target KV. Below we address W1 and W2 with controlled factorial experiments.

W1 & Q1. Separating Native KV Reuse from ORD and On-Policy Rollouts

Comparing native KV under Forward KL (τ=8.41\tau=8.41, 7.36×7.36\times) against JetSpec-N (τ=8.34\tau=8.34, 7.33×7.33\times) understates native KV reuse because OpSpec freezes the pretrained causal-parallel blocks and routes queries into five frozen target WQW_Q/QNorm/RoPE spaces. Output-token KL alone gives weak supervision for multi-layer coordinate alignment, consistent with findings on intermediate and rollout alignment in DistillSpec (Zhou et al., ICLR 2024), EAGLE (Li et al., ICML 2024), HASS (Zhang et al., ICLR 2025), EAGLE-3 (Li et al., NeurIPS 2025), AdaSPEC (Hu et al., NeurIPS 2025), and Draft-OPD (Lei et al., EMNLP 2026).

Two 2×22\times2 factorials (revised Tables 3 and 13) isolate each component cleanly.

  1. Prefix Source ×\times Training Recipe (Table 3). Training reconstructed-KV JetSpec-N with full ORD plus the 180-step on-policy stage yields τ256=8.49\tau_{256}=8.49 (7.43×7.43\times), whereas OpSpec achieves τ256=8.63\tau_{256}=8.63 (7.54×7.54\times), doubling the source advantage from +0.07+0.07 tokens under FKL to +0.14+0.14 tokens (+0.11×+0.11\times speedup) under matched ORD supervision. Within each source, adding ORD plus on-policy branches brings +0.22+0.22 tokens on native KV versus +0.15+0.15 on reconstructed KV.
  2. Objective ×\times State Source on Native KV (Table 13). Final-180-step on-policy histories alone add +0.03+0.03 tokens under FKL (8.41→8.448.41\rightarrow8.44) and +0.04+0.04 under ORD (8.59→8.638.59\rightarrow8.63). At fixed histories, ORD contributes +0.18+0.18 tokens teacher-forced (8.41→8.598.41\rightarrow8.59, NRMSE 0.219→0.1440.219\rightarrow0.144) and +0.19+0.19 tokens on-policy (8.44→8.638.44\rightarrow8.63, NRMSE 0.211→0.1320.211\rightarrow0.132). Under one switchable checkpoint per seed (Appendix A, Table 4), native KV raises τ256\tau_{256} from 8.348.34 to 8.638.63 on branch-specific queries with a +0.17+0.17-token interaction over depth-mean states (p=0.0024p=0.0024).

W2. Distinctions from GliDe and LongSpec

GliDe with a CaPE (Du et al., ICML 2024) pairs autoregressive target-KV cross-attention with confidence-aware expansion, and LongSpec (Yang et al., ACL 2026) combines target cross-attention with bounded sliding-window local draft attention. OpSpec contributes three distinct advances. First, Section 2.2 and Appendix E.4–E.5 formalize prefix non-identifiability, showing rank-rr observed prefix attention rows (r<tr<t) at fixed keys leave a (t−r)dv(t-r)d_v-dimensional unobserved value subspace AΔV=0A\Delta V=0, paired with same-target-query replay (0.1860.186 vs. 0.00020.0002 NRMSE, Appendix D.1, Table 6) and evidence-switching interventions (Table 1). Second, OpSpec merges shared native prefix pages and branch-local ancestors via the shared-denominator softmax log-sum-exp identity (FlashInfer, Ye et al., MLSys 2025) and introduces ORD to supervise prefix readouts, log normalizers, and query directions, cutting readout NRMSE from 0.2260.226 to 0.1510.151 and raising τ\tau from 8.078.07 to 8.278.27 on an independent target-KV drafter (Table 16). Third, OpSpec converts a single-pass causal-parallel tree drafter (JetSpec, Hu et al., 2026) to zero-copy native KV access with zero sequence-linear prefix state.

AuthorsOpenReview ↗

Response to Reviewer FXYK (2/2): Table 4 Reference, JetSpec-N Objective, and All 7 Benchmarks

Q2: Clarifying Table 4 and Its Relationship to DDTree, JetSpec-N, and OpSpec

Thank you for catching the missing inline table citation. Revised Sections 4.2 and 5.1 point to Appendix A and Figure 2 (center).

Regarding the underlying model in Table 4: all four rows of Table 4 come from one single switchable five-layer Qwen3-8B causal-parallel checkpoint per seed (aggregated across three seeds), rather than separate checkpoints for DDTree, JetSpec-N, and OpSpec. During training, each batch routes through either the hidden-reconstructed prefix KV path or the native target KV path with 0.50.5 probability so that a single set of backbone and adapter weights supports both prefix sources. At inference, we evaluate all four combinations of {Reconstructed KV, Native KV} ×\times {Depth-mean query state, Branch-specific query state} without changing any model weights:

  • The Depth mean rows replace each node's path-dependent state at depth kk with the mean state across nodes at depth kk on that same switchable checkpoint. This removes branch-specific query variation within a fixed network; standalone DDTree in Table 2 is a separately evaluated baseline checkpoint rather than one of the four cells in Table 4, and the numerical equality at 7.537.53 is simply the measured outcome of the depth-mean intervention.
  • The Reconstructed + Branch-specific cell (τ=8.34\tau=8.34) corresponds to the JetSpec-N conditioning mode, and Native + Branch-specific (τ=8.63\tau=8.63) corresponds to the OpSpec conditioning mode, evaluated on identical weights to eliminate checkpoint variance.

Q3. Training Objective and Settings for JetSpec-N

JetSpec-N uses the normalized weighted Forward KL objective in Section 3.4, LFKL=T2∑vwvDKL(pvT∥pvD)∑vwv\mathcal L_{\mathrm{FKL}} = T^2 \frac{\sum_v w_v D_{\mathrm{KL}}(p_v^T \mathbin{\Vert} p_v^D)}{\sum_v w_v}, sharing the documented FKL formulation, denominator-normalized node-weighting form, 128,000 Qwen3-8B regenerated examples from the Nemotron V2 and CodeAlpaca mixture, AdamW configuration (lr=3×10−4\text{lr}=3\times10^{-4}, weight decay 0.10.1, BF16 training, 3%3\% warm-up, cosine decay), root anchor, layer normalizations, parameter budget, and 1,000-update teacher-forced schedule with Table 3's Forward-KL-only condition (decoding temperature 11 in Section 5 is a separate inference evaluation, whereas numeric training distillation TT, specific per-node weight values, and per-step batch ordering were not separately logged in the manuscript). Both runs use teacher-forced candidate blocks throughout all 1,000 updates, and revised Table 3 now also trains JetSpec-N with full ORD and the 180-step on-policy stage (τ=8.49\tau=8.49 vs. OpSpec's 8.638.63) for a matched comparison under both training recipes.

Q4: Accepted Lengths and Speedups Across All Seven Benchmarks at B=256B=256

We have added the complete per-benchmark breakdown, including GSM8K and HumanEval, in revised Table 11 (reproduced below).

Benchmark (B=256B=256)JetSpec-N τ\tauOpSpec τ\tauRelative τ\tau GainJetSpec-N SpeedupOpSpec Speedup
MATH-50010.7911.34+5.1%9.66x10.09x
AIME 20259.8410.30+4.7%8.80x9.16x
LiveCodeBench8.819.16+4.0%7.69x7.95x
HumanEval7.808.06+3.3%7.14x7.33x
GSM8K8.648.91+3.1%7.84x8.04x
MBPP7.457.63+2.4%6.75x6.87x
MT-Bench5.956.06+1.8%4.59x4.65x
7-Task Geometric Mean8.348.63+3.5%7.33x7.54x

OpSpec improves point estimates of both accepted length and speedup on every single benchmark with zero regressions. Multi-step mathematical reasoning (MATH-500 +5.1%+5.1\%, AIME 2025 +4.7%+4.7\%, GSM8K +3.1%+3.1\%) and code generation (LiveCodeBench +4.0%+4.0\%, HumanEval +3.3%+3.3\%, MBPP +2.4%+2.4\%) show larger relative gains than open-ended multi-turn chat (MT-Bench +1.8%+1.8\%), consistent with the task-level pattern where sustained reasoning and coding paths more frequently revisit earlier context. We hope these controlled experiments and clarifications thoroughly address your questions.

Thank you to the authors for the clarification. As the authors corrected me in their response to W1, Table 3 does show the 2×2 ablation of prefix source (reconstructed vs. native) × training recipe (forward KL vs. ORD).

My remaining questions are:

(1) From the table, the gains from native KV reuse are smaller than those from ORD (0.07 vs. 0.15 for acceptance length and 0.03 vs. 0.10 for speedup ratio). However, the authors mainly claim their novelty on the native KV reuse method and introduce ORD as a follow-up to native KV reuse. Based on these results, most of the gains contributed by the paper come from ORD, and the submission could better explain why ORD is a better objective and reframe the main text to reflect these empirical gains.

(2) As the authors clarified in their response, their native KV reuse method is methodologically different from other KV reuse methods. The authors mainly compare their proposed native KV reuse method with the reconstruction method. It would be better to highlight the authors' contribution by comparing their proposed KV reuse method with other KV reuse variants trained using the same JetSpec backbone, under forward KL and ORD.

AuthorsOpenReview ↗

Follow-up Response to Reviewer FXYK

Thank you for the constructive follow-up questions.

(1) Role of ORD and Reframing the Main Text

We agree that our results warrant presenting Operator Readout Distillation (ORD) alongside native KV reuse, and we have revised the abstract, Introduction, Section 3.4, and Section 5.3 accordingly. The increase of 0.15 accepted tokens on reconstructed KV (7.33×7.33\times to 7.43×7.43\times speedup) in Table 3 combines ORD with the final 180-step on-policy stage. Holding the history source fixed on native KV (Table 13), ORD adds 0.18 accepted tokens under teacher forcing (8.41→8.598.41\rightarrow 8.59, cutting prefix-readout NRMSE from 0.2190.219 to 0.1440.144) and 0.19 with on-policy histories (8.44→8.638.44\rightarrow 8.63, NRMSE 0.211→0.1320.211\rightarrow 0.132).

Reusing native target KV eliminates the duplicate drafter prefix cache, saving 96.5% of incremental drafter memory at 32K relative to JetSpec-N. Each branch must still learn which prefix tokens to attend to. Forward KL supervises only the output-token distribution. By supervising branch query directions, verified-prefix readouts, and prefix log-normalizers alongside output-token predictions, ORD aligns how each branch reads the target cache.

(2) Same-Backbone Comparison with Target-KV Reuse Variants

As suggested, we adapted GliDe-style target-KV cross-attention and a KVShot-style gated native-KV/hidden-state hybrid onto the same Qwen3-8B JetSpec backbone. The table reports seven-benchmark geometric means at B=256B=256. Speedups are relative to autoregressive decoding. History refers to the final 180 updates.

InterfaceHistoryForward KL τ256\tau_{256}Forward KL speedupORD τ256\tau_{256}ORD speedup
OpSpecTeacher-forced8.417.36×8.597.50×
Target-KV cross-attention (GliDe-style adaptation)Teacher-forced8.367.29×8.517.39×
Gated native-KV and hidden-state hybrid (KVShot-style adaptation)Teacher-forced8.397.27×8.557.38×
OpSpecOn-policy8.447.38×8.637.54×

With teacher-forced histories, OpSpec leads on both τ256\tau_{256} and speedup under Forward KL (8.418.41, 7.36×7.36\times) and ORD (8.598.59, 7.50×7.50\times). ORD improves every KV-reuse interface under teacher forcing. Accepted length increases by 0.15 tokens for cross-attention, 0.16 for the gated hybrid, and 0.18 for OpSpec. We have incorporated this same-backbone comparison and the revised framing into Section 5.3 and Appendix D.7 (Table 13, page 19).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive evaluations. We appreciate your recognition of our same-query diagnostic design (FXYK, 9EkD), evidence-switching experiments (9EkD, p1ud), controlled ablations (9EkD), cache interventions, direct cache-readout supervision, and lightweight frozen-backbone adaptation (p1ud), alongside our incremental memory savings (FXYK, 9EkD, p1ud) and batch-capacity gains (9EkD, p1ud).

We want to further highlight three points:

Native KV reuse and readout distillation are complementary. Reviewers FXYK, 9EkD, and p1ud asked how OpSpec relates to GliDe, LongSpec, and KVShot. OpSpec addresses causal-parallel tree drafting, where sibling branches probe different prefix spans simultaneously. Reusing native KV removes prefix-cache reconstruction error, leaving readout accuracy governed by whether each speculative branch issues the right query into the target's attention space. Operator Readout Distillation (ORD) directly supervises the prefix readout, prefix log-normalizer, and query direction. Training reconstructed-KV JetSpec-N with full ORD plus 180-step on-policy rollouts yields τ256=8.49\tau_{256}=8.49 (7.43×7.43\times), whereas OpSpec reaches 8.638.63 (7.54×7.54\times), doubling the native-KV advantage from +0.07+0.07 to +0.14+0.14 tokens. On native KV, ORD adds +0.18+0.18 teacher-forced and +0.19+0.19 on-policy tokens (reaching 0.1320.132 readout NRMSE with full ORD plus on-policy rollouts), versus +0.03+0.03 to +0.04+0.04 from rollouts alone.

Gains transfer across tasks, scales, drafters, and topologies. On Qwen3-8B, OpSpec improves both accepted length and speedup on every benchmark. On an independent target-KV drafter DKVD_{\mathrm{KV}}, ORD cuts readout NRMSE from 0.2260.226 to 0.1510.151 and lifts throughput by 2.5%2.5\%. On Qwen3-30B-A3B, OpSpec raises seven-task τ256\tau_{256} by 4.1%4.1\% (8.128.12 to 8.458.45), increases 32K throughput by 4.4%4.4\%, and cuts incremental drafter state by 94.8%94.8\%. At active batch 88, speedups hold across 16-token chains (+2.5%+2.5\%) and smaller trees (+3.0%+3.0\% at B=16B=16, +4.4%+4.4\% at peak-throughput B=64B=64).

Execution speed and freed HBM capacity yield distinct serving gains. At 32K context and matched active batch 1010 on H100 GPUs, verification is 49.249.2 ms/round for both methods, while proposal latency falls from 41.741.7 to 39.339.3 ms/round. At this batch size, output throughput rises by 4.8%4.8\% (690.4690.4 to 723.5723.5 tok/s) and total peak HBM falls from 78.678.6 to 69.869.8 GiB. Using those 8.88.8 GiB of saved memory for active batch 1212 (78.978.9 GiB peak) reaches 757.4757.4 tok/s (+9.7%+9.7\%).

We thank all reviewers for prompting these analyses and look forward to the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.