acceptodds
Under review as a conference paper at ICLR 2027

SKIVE: Sparse-KV Importance-scheduled Verification by Expected true positives

Abstract

Speculative decoding (SD) drafts candidate tokens and verifies them in one target forward pass; sparse key–value (KV) cache selection reads only a budgeted subset of the cached context. Verifying the drafted sequence under a sparse cache is lossy in a way the standard acceptance count hides: sparse verification can accept tokens dense verification would reject. We score sparse verification by its true positives: the accept decisions it shares with the dense verifier. From one assumption, that a node's chance of a true positive rises linearly in the importance mass (the selector's scores) the retained blocks keep, we derive an objective for the two decisions a sparse verifier makes: which layers re-select the cache, and how sparse to go. Sparse-KV Importance-scheduled Verification by Expected true positives (SKIVE) applies it to the first, which prior work settles offline by calibrated search. Reusing one selection across all layers costs 15–27 true-positive points; the objective scores a stale selection by the mass it retains, so the best set of re-selecting layers maximizes retained mass, solved exactly by dynamic programming over a table the model builds from its own selections during decoding, and the same solve fixes the anchor count. The schedule needs no calibration data and its only model input is the layer count; on LLaMA-3.1-8B placement matters more than count: the six to eight anchors it chooses reproduce per-layer true positives, whereas six uniformly spaced layers lose 5–8 true-positive points. Against measured costs the objective becomes a throughput function Φ(sparsity, anchor count) that bounds the sparsity and says when sparse verification should, and should not, be used. End to end on LLaMA-3.1-8B with an Arctic-LSTM tree drafter at 64K, the online schedule reaches 5.5× autoregressive (AR) decoding at 4.8% lower ROUGE-L, against 3.4× for per-layer selection and 1.15× for dense SD, and within 3% ROUGE-L of AR with the same sparse cache; it transfers to Qwen3-8B with no calibration, and a fused-kernel SGLang port goes from behind dense tree verification at batch 1 to 52% ahead at batch 8.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, sparse attention, KV cache compression, long-context inference, efficient LLM inference, layer-wise sparsity, test-time adaptation, dynamic programming

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WSLKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies SKIVE, a lossy speculative tree verification method that reduces cost by selecting a subset of KV-cache blocks and target-model layers to guide KV-block selection. The authors measure the agreement with dense verification through true positives (TP), separating them from false accepts that ordinary acceptance counts include. From this viewpoint, SKIVE uses its "importance mass" as a surrogate for expected TP, motivated by an affine approximation to sparse-acceptance probabilities. Then, the choice of KV blocks aims to maximize the surrogate expected TP during decoding online by using dynamic programming.

Critical strengths
  • Table 2 separates the effects of reusing selections across tree nodes and across layers. Scoring only the root query at every layer improves TP relative to reusing layer 0's selection, whereas aggregating node queries provides little additional improvement. This distinction provides useful guidance on where to focus selection computation in the tested settings.
  • The online scheduler seems to improve throughput over per-layer selection in the reported implementation.
Critical weaknesses

W1. I cannot understand the derivation of the surrogate for expected TP in Equation (1).

  • My first concern is the starting equality in Appendix B.1, before the affine approximation is introduced. Although a true positive requires both dense and sparse acceptance, the appendix writes E[TP](S)=∑vπvPr⁡(Av∣S),\mathbb{E}[\mathrm{TP}] (S)=\sum_v\pi_v\Pr(A_v\mid S), where AvA_v denotes sparse acceptance and πv\pi_v is the normalized, depth-discounted drafter path prior. If DvD_v denotes dense acceptance, with both events including the necessary path conditions, the identity for the expected TP count is

    =\sum_v\Pr(D_v\mid S)\Pr(A_v\mid D_v,S).$$ The drafter prior is not established to equal the dense-acceptance probability, and the sparse-acceptance term omits conditioning on dense acceptance. Section 4.1 explicitly describes the drafter weights as a free prior, which can motivate a surrogate, but does not justify these replacements. Thus, the starting equality does not hold in general, and the affine assumption alone does not establish the identification with expected TP.
  • I acknowledge that Table 6 provides empirical support for retaining the uncorrected node weights: the tested reweightings do not improve the measured objective. However, I am not convinced that this validates the affine probability-versus-mass approximation, since the table does not directly measure that relationship or test a common slope and intercept across nodes. Assuming this relationship does make the within-layer top-K argument valid for the proposed weighted objective, but it does not resolve the preceding gap between that objective and expected TP.

  • Even granting the approximation above, I do not see how the equally weighted layer sum follows. TP is defined through final verification decisions, and the paper does not establish its decomposition into layer contributions. Even if those contributions could be added, substituting Equation 2 into Equation 1 gives

    \approx \lambda_0+\lambda\\,m_{\ell}(S_{\ell})\\,\Gamma[\ell,a].$$ The coefficient of the normalized mass therefore depends on the layer's own retained mass, so equal weighting requires further justification. Appendix C.2 tests a late-layer weighting that helps Qwen3-8B but hurts LLaMA-3.1-8B; this informs the design choice without supplying the missing derivation. Upstream selection changes can also alter downstream queries, whereas the DP treats the table measured in full-scoring rounds as fixed. The solver is exact for Equation 3, but this does not establish exact optimization of expected TP. Figure 2 and Table 9 already provide some evidence that retained mass is useful for schedule selection, and I would treat Equation 3 as an empirical surrogate on that basis. A direct comparison of objective scores and measured TP across alternative schedules, with queries recomputed under each schedule, would further clarify its reliability.

W2. The benefit over the closest prior anchor-selection rule remains unclear.

  • I am not convinced that the improvement over the row labeled “Root reuse (Wang)” measures the improvement over Wang et al.'s anchor-selection rule. That row computes selections once at layer 0, whereas Wang et al., Section 3.3, Equations 7–9 select multiple anchors using adjacent-layer Jaccard scores on calibration data and reuse the nearest preceding anchor. Table 9 does compare SKIVE with uniform schedules and TidalDecode's published anchor placements transplanted into the sparse verifier, showing that anchor placement matters in the tested settings. These comparisons do not evaluate Wang et al.'s calibrated multi-anchor rule, which the paper itself identifies as closely related. I would like to see a comparison with that rule at the same KV budget and comparable measured selection cost, accounting for both calibration and online measurement costs. This would clarify whether SKIVE improves the quality/throughput tradeoff or achieves a comparable tradeoff without offline calibration.
Questions for authors

Q1. Comparison is needed to assess the contribution.

  • How does the online controller compare with Wang et al.'s calibrated multi-anchor rule at the same KV budget and comparable measured selection cost, including the calibration/setup cost? I am interested in whether the benefit is better throughput and quality, or comparable performance without offline calibration.

Q2. Clarification of the quality constraint.

  • Table 3 chooses sparsity using ROUGE-L on the same GovReport documents that are reported, and Table 14 gives a 90% equivalence bound of approximately ±10.8%\pm10.8\% for the 64K online configuration without refresh. The appendix explicitly reports this limitation, and the arXiv transfer results provide positive evidence of transfer. Is there an evaluation in which sparsity was fixed independently of the evaluated documents, and does the uncertainty support the stated 5% tolerance? Otherwise, the quality floor should be described as an in-sample operating-point criterion, rather than evidence that degradation is bounded by 5%.
2 comments
AuthorsOpenReview ↗

Response to Reviewer WSLK (1/2): the derivation

Thank you for the careful reading of the derivation. You are right on each point. The method and all results are unchanged; what changed is what the paper claims, and how it supports it.

1. The starting equality was an approximation. §4.1 and App. B.1 now start from the exact identity you wrote,

E[TP]=∑vPr⁡(Dv)⋅Pr⁡(Sv∣Dv)(Eq. 6)\mathbb{E}[\mathrm{TP}] = \sum_v \Pr(D_v) \cdot \Pr(S_v \mid D_v) \quad \text{(Eq. 6)}

where DvD_v and SvS_v mean that the dense and the sparse verifier accept node vv. The paper then names the two substitutions: the drafter's prior stands in for the first factor, and the second factor is modelled as rising with the mass the selected blocks retain for vv. Eq. 1 and Eq. 3 are now called surrogates, as you suggest, and the solver is described as exact for Eq. 3, not for expected TP.

2. The schedule-level comparison you asked for (App. B.1, Fig. 5). We ran fourteen schedules, each under its own selections, so the queries are recomputed under each schedule: our solver's anchors at six counts from 3 to 14, the calibrated anchors of Wang et al. at four counts, two hand-made schedules, five uniform anchors, and every-layer selection. Ranking them by Eq. 3 and by measured TP gives nearly the same order. The Spearman rank correlation is 0.81, 0.92, 0.88 and 0.90 in the four settings (16K and 32K, s = 0.90 and 0.95; all p < 0.001; 1 means identical order).

This test compares the objective with the final verification outcome, so it covers your points about the layer sum. It does not assume that TP decomposes over layers, and it runs every schedule with its own reused selections upstream. You are also right that equal layer weights do not follow from Eq. 1. §4.3 now states them as a design choice.

3. The per-node test of the linear step (App. B.1, Fig. 4). We measured acceptance against retained mass with the node held fixed: the same node, in the same round, at the same budget, under four different selections.

  • What holds. The selection that keeps more of a node's mass accepts it more often in all 24 pairwise comparisons.
  • What does not. Slope and intercept are not common across nodes. The slope is steeper for deeper nodes and at higher sparsity.
  • Why the method is unaffected. It uses only the ordering Eq. 1 gives, never its coefficients: selection inside a layer is a top-K, and the schedule is an arg max. The node weights barely matter either: scoring the root query alone is within 0.3 TP points of the full aggregate (Table 2).
AuthorsOpenReview ↗

Response to Reviewer WSLK (2/2): comparison with Wang et al., the quality constraint

W2 and Q1. The calibrated multi-anchor rule of Wang et al.

We implemented their rule: layer 0 plus the layers whose selection overlaps least with the previous layer's on calibration documents ("Jaccard anchors" below). We ran it at the same KV budget and the same number of anchors as ours (Table 14). Their paper leaves the number of anchors as a free parameter and gives none for LLaMA-3.1-8B, so we gave it ours.

The answer to your question: comparable quality and throughput, without offline calibration.

Change in TP points against every-layer selection (s = 0.95, six anchors)16K32K64K
Jaccard anchors as published (scoring the root query only)−2.2†−0.7−0.1
Jaccard anchors, tree nodes scored our way−1.0*−0.4−0.4
Our solver's six anchors−0.1−0.3−0.1
Online controller (chooses 6–8 anchors itself)−0.8−0.10.0

(* p < 0.05, † p < 0.01.) End to end at 64K with six anchors, the Jaccard anchors reach 5.92× AR at −3.3% ROUGE-L and our solver's anchors 5.96× at −3.4%.

What ours adds:

  • No calibration. No calibration documents and no calibration pass.
  • It chooses the number of anchors (Eq. 4). Their rule has to be given it.
  • It keeps adapting while decoding. This costs about 3% of speed at 64K: 5.50× AR for the controller against 5.65× for anchors fixed offline, in the same run.

The two rules agree because they detect the same thing, the layers where a reused selection goes stale (Fig. 2a): theirs through set overlap, ours through retained mass. When two schedules differ by one TP point or less, neither rule is reliably ahead.

The row label. The row formerly labelled "Root reuse (Wang)" selects once at layer 0. It is now labelled "Root reuse (layer 0)", and §5.1 states which rows measure the two published variants: "Root query, every layer" (Table 2) and "Jaccard anchors" (Table 14).

Q2. The quality constraint.

The 5% in Table 3 is the threshold of the rule that picked each sparsity: the sparsest setting whose loss on those fifty documents is within 5%. You are right that this rule was applied to the same documents the table reports. The caption now calls it an in-sample criterion, and we evaluated the same sparsities on documents that played no part in choosing them (App. B.4, Table 12). ROUGE-L change against AR for the online controller:

4K16K32K64K
Fifty held-out GovReport documents−5.3%−7.5%*−3.5%too few documents
Held-out and Table 3's documents together (100)−5.0%*−5.1%*−4.1%*
Fifty held-out arXiv documents−0.8%
Sparsity chosen on arXiv, scored on GovReport−5.6%*−5.4%*−4.7%−4.8%
  • The loss at these sparsities is about 5% of AR's ROUGE-L, and the paper states it as that.
  • The uncertainty. The 90% equivalence bound is ±5.9% on the held-out arXiv documents at 64K and ±7.5–8.4% on the hundred GovReport documents. So the data rule out a loss larger than 6–8%. They do not make 5% a guarantee, and Table 3 uses it only as the selection threshold.
  • Where the loss comes from: the sparse cache, not our schedule. Every-layer selection loses 5.1–5.9% on the same hundred documents. At 16K, AR reading the same sparse cache, with no speculation at all, loses 7.3%. On the held-out documents the controller's ROUGE-L is within two standard errors of every-layer selection's (they differ by 1.2% or less; two standard errors is 3%), while the controller is 1.2–1.3× faster.

Presentation. §4.1 and App. B.1 are reorganized into what is assumed, what is tested and what the method needs. We would value your reading of them.

Reviewer_PocCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SKIVE, a sparse KV-cache verification framework for speculative decoding in long-context LLM inference. The paper identifies that speculative decoding and sparse verification serves as complementary method, but sparse verification can introduce false acceptances that are hidden by the conventional speculative decoding acceptance metric. To address this issue, SKIVE proposes expected true positives as the optimization objective, where the retained importance mass of selected KV blocks is used as a proxy for verification fidelity. Based on this objective, SKIVE introduces an online layer scheduling algorithm that selects anchor layers for KV re-selection and reuses selections across other layers. It further derives a cost-aware operating point to determine the sparsity level. Experiments on multiple models demonstrate improved throughput over dense speculative decoding and previous sparse verification approaches.

Critical strengths
  1. The paper identifies an important issue in combining sparse KV and speculative decoding, and proposed TP/FP as a more meaningful evaluation metric.
  2. The paper proposed layer-wise KV selection reuse as an optimization problem. Instead of relying on manually selected anchor layers or sparsity configurations, SKIVE derives the layer schedule and operating point from its objective and online measurements.
  3. The paper provides a relatively comprehensive system evaluation, including different models, drafters, context lengths, and an SGLang implementation.
Critical weaknesses
  1. The most important concern would be the distribution shift between dense and sparse verification is not sufficiently characterized. But the evaluation mainly focuses on output similarity metrics (e.g., ROUGE-L), it remains unclear whether this kind of shift would affect downstream performance.
  2. The shared KV selection across speculative nodes introduces a potential mismatch between node-level query distributions, and this mismatch may depend on the size and structure of the speculative tree.
  3. The paper extensively analyzes different schedules in the HF experiments, the SGLang production evaluation does not provide a detailed ablation study of the proposed components
Questions for authors

Q1: Could the authors provide more analysis on the relationship between dense-sparse verification discrepancy (e.g., TP/FP behavior or trajectory deviation) and downstream task performance?

Q2: Could the authors analyze how SKIVE behaves under different tree configurations, such as different draft widths, depths?

Q3: Could the authors provide a component-wise analysis on the production stack to quantify the contribution of sparse verification, anchor scheduling, and other system-level optimizations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PocC (1/2): verification discrepancy and downstream performance

Thank you for the review. This comment answers W1 and Q1.

In short

  1. We added five downstream tasks, which makes ten beyond summarization. Our method, the online controller, is within noise of AR on sixteen of twenty task cells. The clear cost is on retrieval-style tasks.
  2. What predicts the downstream loss is the false-positive (FP) rate: how often the sparse verifier accepts a draft token that the dense verifier would reject.
  3. How far the output text drifts from AR's text does not predict it.

Downstream tasks (App. D.4, Tables 18–19). Multi-hop QA (HotpotQA, 2WikiMQA, MuSiQue), single-document QA (NarrativeQA, MultiFieldQA), code completion (LCC, RepoBench-P), question classification (TREC), passage retrieval and needle retrieval. Five are new (Table 19). A cell is one task at one context length and sparsity.

  • Sixteen of twenty LongBench cells are within noise of AR.
  • The four exceptions are mostly retrieval-style: passage retrieval at both sparsities we ran (−6.3 and −8.2 points of accuracy, against AR's 97.6%), TREC at the higher sparsity (−11 points of accuracy; −3, not significant, at the lower one), and one multi-hop QA cell (MuSiQue at 8K, −0.027 F1).
  • Needle retrieval: our method finds the needle in 92–100% of cases, AR in 100%.
  • Root reuse, the earlier schedule that chooses blocks once at layer 0, loses significantly on seventeen of the twenty cells and finds no needle at 32K and 64K.

What predicts the loss: the FP rate (App. E, Fig. 12). Each point in Fig. 12 is one schedule at one context length: its FP rate against its change in quality relative to AR.

  • The higher the FP rate, the larger the loss. The Pearson correlation is −0.96 on arXiv (twelve points), −0.87 on GovReport and −0.74 on QA (token-F1).
  • In absolute terms. A schedule with an FP rate of 18–19% (reuse of layer 0) loses 8–15% ROUGE-L. Schedules with FP rates of 2–5% lose at most 6%.
  • Its limit. Among schedules with FP rates below 6%, the remaining differences are too small for the FP rate to rank.

This is why the method maximizes agreement with the dense verifier.

What does not predict it: token-level deviation from AR (App. E). An output can be scored against the gold summary, as in Table 3, or against AR's own output. Against AR's output, dense SD scores 0.96 (FP16 noise) and every sparse method reaches 0.33–0.55. Against the gold summary, the online controller scores within noise of AR at every point of Table 3. Both can be true because AR itself scores only about 0.2 against the gold summary: a different summary can be just as good. Sparse outputs are different because one different accept decision changes all the text after it. So the score against AR's output says when the first difference happened, not how good the output is.

What per-round agreement misses. Errors that build up in the cache over a long output. At 512 generated tokens the controller is 0.5–2.4% of AR's ROUGE-L below every-layer selection, the schedule in which all 32 layers choose their own blocks (Table 13). It is 1.2–1.7× faster.

AuthorsOpenReview ↗

Response to Reviewer PocC (2/2): tree configuration, components on the production stack

This comment answers W2, Q2 and Q3.

In short

  1. The mismatch you describe is real, and we measured it. It grows with a node's depth in the tree. It does not grow with the size of the tree.
  2. Across trees of 30 to 100 nodes and expansion widths of 4 to 20, the online controller stays within 0.4 TP points of every-layer selection.
  3. On the production stack, sparse verification alone gives no gain on speed. The schedule gives all of it.

Two terms. Every-layer selection lets all 32 layers choose their own cache blocks. A TP point is one percentage point of the dense verifier's accepted draft tokens that the sparse verifier also accepts.

W2 and Q2. Tree configuration (App. B.2, Tables 8–10)

Your concern, as we understand it. Each round the drafter proposes a tree of candidate tokens, the nodes. Every node has its own query, but all nodes share one set of cache blocks. A node whose query differs from the others may be served poorly by that shared set, and a larger tree might make this worse.

  • How we measured it. For each node we take the mass the shared set keeps for it, and divide by the mass its own best set of the same size would keep. A ratio of 1 means the shared set serves the node as well as its own choice would.
  • Deeper nodes are served slightly worse. The ratio is 0.99 at the root, 0.95–0.96 at depth 1 and 0.91 at depth 5 (Fig. 7).
  • A larger tree does not make it worse. At a fixed depth, going from 60 to 100 nodes changes the ratio by +0.0004 (90% interval −0.0003 to +0.0011).
  • A direct test (Table 8). We took one 100-node tree and built the shared set three ways: from all 100 nodes, from the best 60, and from the best 30. We then verified the same 100 nodes with each set. The TP rate is 0.9453, 0.9451 and 0.9453 (last row of Table 8), so adding nodes to the mix does not hurt the set.
  • A larger tree changes the candidates, not the shared set. The TP rate at 32K is 0.955 with 30 nodes and 0.946 with 100. The nodes a larger tree adds are the drafter's long shots, and they lie further ahead in the draft: on average three tokens past the committed text, against 1.7 for the first 30 nodes. A node that far ahead counts only if every node before it on its path was accepted. On its own decision, too, the sparse verifier agrees with the dense one less often: 94–95% against 97%.

Across tree shapes (Table 9: trees of 30 to 100 nodes, each node expanded into its 4 to 20 most likely next tokens, at 32K and 64K, s = 0.95), change in TP points against every-layer selection:

ScheduleRange over all tree shapes
Online controllerwithin 0.4, none significant
Six uniform anchors−4.7 to −6.0
Layer 0's choice reused at every layer−13.5 to −18.6
  • On depth. The tree keeps its highest-scoring nodes, so a depth limit barely changes it. We therefore varied the number of nodes and how many next tokens each node is expanded into.
  • End to end with the Arctic LSTM drafter (Table 10). As the tree grows from 28 to 110 nodes, dense SD falls from 1.73× to 0.96× AR at 32K. The controller holds 3.5–3.8× at 32K and 5.1–6.0× at 64K, at a ROUGE-L loss of 3.7–5.2%.

Q3. Components on the production stack (App. F, Table 22). One stack, one H200 job, 64K. Throughput relative to dense SD, that is, speculative decoding reading the full cache (above 1 is faster):

Batch size1248
Sparse kernel, every layer selecting0.840.871.000.99
+ schedule (six anchors)1.021.151.331.53
+ online controller0.981.111.271.38
  • Sparse verification alone: no gain. Selecting blocks at all 32 layers costs about what the sparse read saves.
  • The schedule: all of the gain.
  • The cache of block statistics: at most 3%.
  • The online controller: runs at 0.95 of the throughput of six anchors placed offline, on average. That is the price of choosing anchors online (Table 23).
Reviewer_jVWZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SKIVE, a framework for scheduling sparse key–value (KV) cache selection in long-context speculative decoding. It measures true positives (TP)—draft tokens accepted by both sparse and dense verification—to expose false acceptances hidden by acceptance length. Using retained importance mass as a surrogate for expected TP, it selects anchor layers by dynamic programming over a table updated during decoding. A cost model relates sparsity and anchor count to verification throughput. Experiments on LLaMA-3.1-8B and Qwen3-8B, with a separate fixed-schedule SGLang port, show efficiency gains with task-dependent quality losses.

Critical strengths
  1. Makes false acceptances visible and gives scheduling a concrete objective. Acceptance length alone can overstate the effectiveness of a sparse verifier because it includes tokens dense verification would reject. The TP objective addresses this problem, and the retained-mass surrogate yields a tractable layer-scheduling problem. The online controller chooses anchor placement without a separate offline search on calibration data.

  2. Controlled comparisons support layer scheduling. The experiments separate selection reuse across tree nodes from reuse across layers. Table 2 and the anchor-count studies show that online scheduling stays close to per-layer TP while scoring far fewer layers. Table 3 reports substantial long-context summarization speedups, and the SGLang port shows a throughput advantage over dense tree verification as batch size grows.

Critical weaknesses
  1. Quality-based operating points need independent evaluation. The dynamic program optimizes a retained-mass surrogate. Appendix B.1 tests nonlinear weight corrections, but Appendix E shows that per-round TP misses errors from sparsely written KV states accumulating across rounds. Table 3 selects sparsity using ROUGE-L on the same 50 GovReport documents used for evaluation, with 128 generated tokens. Non-significant declines do not establish tight quality equivalence: Table 14's 64K online row has a 90% equivalence bound of ±10.8%. The resulting quality–throughput trade-off needs evaluation after fixing operating points independently, particularly for longer outputs.

  2. The serving results do not test the complete online controller. The main throughput study uses LLaMA-3.1-8B on two summarization datasets, with additional models and tasks in Appendix D. On Qwen3-8B, uniform anchors perform similarly, so the placement advantage is model-dependent. More importantly, the SGLang port uses six fixed anchors rather than online count and placement (§6). Appendix F measures quality only at batch size one. These results support scheduled sparse verification, but leave the full controller's cost, benefit, and output fidelity under concurrent serving untested in that stack.

Questions for authors
  1. Independent operating-point selection. Given that Table 3 selects sparsity on the evaluated GovReport documents, can you separate selection from evaluation using held-out documents or document-level cross-validation where the 64K population is small? Using existing results where possible, how does SKIVE compare with per-layer selection at matched output quality? Longer-output measurements would also help assess the sparse-write errors documented in Appendix E.

  2. The full controller in SGLang. Can you evaluate online anchor count and placement against the fixed and uniform schedules in the same serving stack, including periodic full-scoring and table-update costs? Please also report output quality at batch sizes above one; the current throughput and acceptance-length measurements do not establish fidelity in those settings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jVWZ (1/2): held-out documents, matched quality, longer outputs

Thank you for the review. This comment answers your first question.

In short

  1. On held-out documents the controller loses about 5% of AR's ROUGE-L, the size Table 3 shows except at 16K (7.5% against 2.4%).
  2. The loss comes from the sparse cache, not from the schedule: every-layer selection ("per-layer selection" in your review) loses as much or more.
  3. At matched quality the controller is 1.2–1.6× faster than every-layer selection.
  4. The loss does not build up with output length.

Held-out documents (App. B.4, Table 12). Table 3 chose each sparsity on fifty GovReport documents. We ran the next fifty GovReport documents at the same sparsities. ROUGE-L change against AR:

4K16K32K
Controller, Table 3's fifty documents−4.7%−2.4%−4.7%
Controller, fifty held-out documents−5.3%−7.5%*−3.5%
Controller, all hundred−5.0%*−5.1%*−4.1%*
Every-layer selection, all hundred−5.3%†−5.9%†−5.1%*

(* p < 0.05, † p < 0.01.)

  • 64K. GovReport has only 57 long-enough documents. Fifty held-out arXiv documents give −0.8%, and re-choosing the sparsity with each GovReport document left out in turn gives −4.4% on the left-out documents.
  • Where the loss comes from. Every-layer selection loses as much or more (last row), so the schedule is not the cause. At 16K, where the held-out loss is largest, AR reading the same sparse cache with no speculation at all loses 7.3%.

Matched quality against every-layer selection. On the held-out documents, in the unit of the table above (% of AR's ROUGE-L):

4K16K32K64K (arXiv)
Controller above (+) or below (−) every-layer selection−0.1%+0.8%+1.2%−1.9%
Two standard errors of that difference3%3%3%5%
Controller's speed relative to every-layer selection's1.22×1.25×1.34×1.62×

Every difference is within two standard errors, so the two already match in quality while the controller is 1.2–1.6× faster. At 64K we also matched quality the other way: when the controller keeps twice as many cache blocks (10% instead of 5%), it reaches 4.64× AR at −2.3%, against 3.44× at −1.7% for every-layer selection. That is 1.35× faster at equal quality.

Longer outputs (Table 13). ROUGE-L change against AR at 512 generated tokens. Each cell gives Table 3's documents first, then the held-out documents:

ContextControllerEvery-layer selection
4K−9.8%, −7.2%−7.5%, −5.0%
16K−4.9%, −9.4%−3.2%, −7.0%
32K−4.2%, −4.0%−3.1%, −2.8%
64K−1.2% (at 6.0× AR)−0.7%
  • The loss does not build up with length. From 128 to 512 generated tokens it shrinks at 64K (to 1.2%), stays level at 32K, and grows by at most 3 points at 4K and 16K.
  • Short contexts lose more (5–10%). Only generated tokens are written under sparse attention, and they are 12.5% of the cache at 4K against under 1% at 64K.
  • Reuse has a small cost on long outputs. At 512 tokens the controller is 0.5–2.4% of AR's ROUGE-L below every-layer selection. At 128 tokens the two are equal. In exchange the controller is 1.2–1.7× faster.
AuthorsOpenReview ↗

Response to Reviewer jVWZ (2/2): the full controller in SGLang, model dependence

This comment answers your second question and the model dependence you noted.

In short

  1. The full online controller now runs in SGLang. At 64K it is faster than dense SD from batch 2 upward, and 38% faster at batch 8.
  2. Choosing anchors online instead of fixing them offline costs about 5% of throughput.
  3. Output quality is now measured at every batch size up to 8. No cell of our method differs significantly from AR.
  4. The placement advantage does depend on the model, as you say. We mapped it on nine models. The controller is within 1.9 TP points of every-layer selection on all of them.

The full controller in SGLang (§6, App. F, Table 22). It now runs in the serving path with its full-scoring rounds, table updates and exact solve, measured in one H200 job together with the other schedules. Throughput relative to dense SD (above 1 is faster):

Batch size1248
64K, online controller0.981.111.271.38
64K, six anchors placed offline1.021.151.331.53
64K, six uniformly spaced anchors1.081.171.341.51
64K, every-layer selection0.840.871.000.99
32K, online controller0.880.971.031.32
  • Benefit. Every-layer selection never beats dense SD. The schedule is what makes sparse verification pay in this stack.
  • Cost of choosing online. The six anchors placed offline come from the same solver, run before decoding, so that row against the controller's row is offline against online. The controller runs at 0.95 of their throughput on average. At batch 8 we measured the two things it does extra (Table 23):
    • It chooses seven or eight anchors instead of six. Eight anchors placed offline run at 0.92 ± 0.04 of the throughput of six at 64K, and at 0.99 ± 0.08 at 32K.
    • It runs three full-scoring rounds per request. Held at six anchors, the controller runs at 0.92 ± 0.03 of six placed offline at 64K, and at 1.00 ± 0.08 at 32K.
  • Uniform anchors are as fast, as expected: speed depends on how many layers select, not on which. Which layers select decides quality; see below.
  • Quality at batch sizes above 1 (Table 24). We paired ROUGE-L against AR at the same batch size, up to batch 8. No six-anchor or controller cell differs significantly from AR. The 90% equivalence bounds run from ±5.5% (64K, batch 8) to ±17.6% (32K, batch 2).

Model dependence (App. D.3, Table 17). We compared the schedules on nine models from three families: four of the LLaMA lineage (LLaMA-3.1-8B, LLaMA-3.2-3B, DeepSeek-R1-Distill-LLaMA-8B and a long-context fine-tune of LLaMA-3-8B), Mistral-7B, and Qwen3 at 1.7B, 4B, 8B and 14B. A trained drafter exists for only a few of these models, so we measured all nine without one: each model drafts its own next four tokens, and the sparse verifier is scored against the dense one on those tokens. On LLaMA-3.1-8B at 16K this reproduces the numbers measured with the EAGLE3 drafter (six uniform anchors lose 8.0 TP points against 7.5; the controller 1.0 against 0.8).

Change in TP points against every-layer selection (16K and 32K, s = 0.95, 30 documents). A TP point is one percentage point of the dense verifier's accepted draft tokens that the sparse verifier also accepts.

LLaMA lineage (4 models)Mistral-7B and Qwen3 (5 models)
Online controller−0.3 to −1.7+0.1 to −1.9
Six uniform anchors−5.1 to −13.5−0.7 to −3.0
Eight uniform anchors−1.0 to −2.6−0.6 to −1.9
Layer 0's choice reused at every layer−20 to −35−28 to −34
  • Reuse across layers is the problem on every model. It loses 20–35 TP points on all nine.
  • Placement matters on the LLaMA lineage, not on Mistral or Qwen3. On the LLaMA lineage the controller beats eight uniform anchors in seven of eight cells. On Mistral and Qwen3 uniform anchors do as well.
  • The controller handles both cases with the number of layers as its only input, while scoring only 25–42% of the layers. A user does not need to know which kind of model they have.

Thank you for your rebuttal, I have read your rebuttal and I choose to keep my score unchanged.

Thank you for taking the time to read our rebuttal and reconsider our work. We appreciate your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision (to all reviewers)

Thank you for three careful reviews. The revised PDF contains every experiment you asked for. Section and table numbers below refer to it.

Terms used in all our replies

  • Every-layer selection: sparse verification in which all 32 layers choose their own cache blocks. The most careful schedule, and the slowest.
  • Online controller: our method. Only 7–8 layers, the anchors, choose blocks. Every other layer reuses the nearest anchor's choice. The controller picks the anchors itself while decoding.
  • Dense SD: speculative decoding that reads the full cache.
  • TP rate: of the draft tokens the dense verifier accepts, the share the sparse verifier also accepts. A TP point is one percentage point of it.

What the new experiments show

  1. The controller is as accurate as every-layer selection, and faster. We ran both on fifty held-out documents per context: documents never used to choose the sparsity or any other setting (GovReport at 4K, 16K and 32K; arXiv at 64K, where GovReport has too few long documents).
    • 4K, 16K and 32K. The two ROUGE-L scores are within two standard errors of each other: they differ by 1.2% or less, and two standard errors is 3%. The controller is 1.2–1.3× faster.
    • 64K. The controller is 1.6× faster. Its ROUGE-L is 1.9% lower, again within two standard errors (5%). When it keeps twice as many cache blocks (10% instead of 5%), so that its score matches every-layer selection's, it is still 1.35× faster (App. B.4).
  2. Our objective predicts measured true positives. We measured the TP rate of fourteen schedules: our solver's anchors at six counts, the calibrated anchors of Wang et al. at four counts, two hand-made schedules, uniform anchors and every-layer selection. Ranking them by our objective (Eq. 3) and by measured TP rate gives nearly the same order: Spearman rank correlation 0.81–0.92, where 1 means identical order (App. B.1). This is the test WSLK proposed.
  3. The controller stays accurate on other draft trees and other models. With draft trees of 30 to 100 nodes it is within 0.4 TP points of every-layer selection (App. B.2). On nine models from three families, given only the number of layers, it is within 1.9 TP points, 0.9 on average. Reusing layer 0's choice at every layer loses 20–35 TP points on all nine (App. D.3).
  4. The full controller runs in a serving stack. In SGLang at 64K and batch 8 it is 38% faster than dense SD. Its ROUGE-L is not significantly different from AR's at any batch size up to 8 (App. F).
  5. It matches the calibrated anchors of Wang et al. without calibration. Their rule needs calibration documents and must be told how many anchors to use. Ours needs neither. Given the same number of anchors and the same way of scoring tree nodes, the two are within one TP point of each other, and end to end at 64K they reach the same speed and ROUGE-L (App. C.1).

What we changed in the claims

  • Derivation (WSLK). The submitted derivation wrote expected TP as an equality. It is an approximation. §4.1 now starts from the exact identity, names the two substitutions, and calls Eq. 1 and Eq. 3 surrogates. The method and all results are unchanged, and the surrogate is now tested directly (item 2 above).
  • Quality (jVWZ, WSLK). Table 3 picks each sparsity by a rule applied to the documents it reports, and its caption now labels this an in-sample criterion. On held-out documents the loss against AR at these sparsities is about 5% of AR's ROUGE-L. The sparse cache causes it, not the schedule: every-layer selection loses as much or more on the same documents.
  • Long outputs (jVWZ). The loss does not build up with output length. From 128 to 512 generated tokens it shrinks at 64K (to 1.2%, at 6.0× AR), stays level at 32K (about 4%), and is 5–10% at 4K and 16K.

Two faults in the submitted SGLang port, both fixed. Porting the controller exposed them, and every number in App. F is re-measured with the fixes.

  1. At batch 1 the port ignored the schedule and selected at every layer. The submitted batch-1 cell, 19% behind dense SD, was therefore an every-layer result. With the schedule applied, six anchors run at 1.02× the speed of dense SD (±0.14).
  2. At batch 2 and above, the cache of block statistics could return another request's statistics.

The batched results are unchanged within noise: six anchors against dense SD at 64K and batch 2, 4, 8 were 1.14×, 1.38×, 1.52× and are 1.15×, 1.33×, 1.53×.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.