- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies SKIVE, a lossy speculative tree verification method that reduces cost by selecting a subset of KV-cache blocks and target-model layers to guide KV-block selection. The authors measure the agreement with dense verification through true positives (TP), separating them from false accepts that ordinary acceptance counts include. From this viewpoint, SKIVE uses its "importance mass" as a surrogate for expected TP, motivated by an affine approximation to sparse-acceptance probabilities. Then, the choice of KV blocks aims to maximize the surrogate expected TP during decoding online by using dynamic programming.
Critical strengths
- Table 2 separates the effects of reusing selections across tree nodes and across layers. Scoring only the root query at every layer improves TP relative to reusing layer 0's selection, whereas aggregating node queries provides little additional improvement. This distinction provides useful guidance on where to focus selection computation in the tested settings.
- The online scheduler seems to improve throughput over per-layer selection in the reported implementation.
Critical weaknesses
W1. I cannot understand the derivation of the surrogate for expected TP in Equation (1).
-
My first concern is the starting equality in Appendix B.1, before the affine approximation is introduced. Although a true positive requires both dense and sparse acceptance, the appendix writes where denotes sparse acceptance and is the normalized, depth-discounted drafter path prior. If denotes dense acceptance, with both events including the necessary path conditions, the identity for the expected TP count is
=\sum_v\Pr(D_v\mid S)\Pr(A_v\mid D_v,S).$$ The drafter prior is not established to equal the dense-acceptance probability, and the sparse-acceptance term omits conditioning on dense acceptance. Section 4.1 explicitly describes the drafter weights as a free prior, which can motivate a surrogate, but does not justify these replacements. Thus, the starting equality does not hold in general, and the affine assumption alone does not establish the identification with expected TP. -
I acknowledge that Table 6 provides empirical support for retaining the uncorrected node weights: the tested reweightings do not improve the measured objective. However, I am not convinced that this validates the affine probability-versus-mass approximation, since the table does not directly measure that relationship or test a common slope and intercept across nodes. Assuming this relationship does make the within-layer top-K argument valid for the proposed weighted objective, but it does not resolve the preceding gap between that objective and expected TP.
-
Even granting the approximation above, I do not see how the equally weighted layer sum follows. TP is defined through final verification decisions, and the paper does not establish its decomposition into layer contributions. Even if those contributions could be added, substituting Equation 2 into Equation 1 gives
\approx \lambda_0+\lambda\\,m_{\ell}(S_{\ell})\\,\Gamma[\ell,a].$$ The coefficient of the normalized mass therefore depends on the layer's own retained mass, so equal weighting requires further justification. Appendix C.2 tests a late-layer weighting that helps Qwen3-8B but hurts LLaMA-3.1-8B; this informs the design choice without supplying the missing derivation. Upstream selection changes can also alter downstream queries, whereas the DP treats the table measured in full-scoring rounds as fixed. The solver is exact for Equation 3, but this does not establish exact optimization of expected TP. Figure 2 and Table 9 already provide some evidence that retained mass is useful for schedule selection, and I would treat Equation 3 as an empirical surrogate on that basis. A direct comparison of objective scores and measured TP across alternative schedules, with queries recomputed under each schedule, would further clarify its reliability.
W2. The benefit over the closest prior anchor-selection rule remains unclear.
- I am not convinced that the improvement over the row labeled “Root reuse (Wang)” measures the improvement over Wang et al.'s anchor-selection rule. That row computes selections once at layer 0, whereas Wang et al., Section 3.3, Equations 7–9 select multiple anchors using adjacent-layer Jaccard scores on calibration data and reuse the nearest preceding anchor. Table 9 does compare SKIVE with uniform schedules and TidalDecode's published anchor placements transplanted into the sparse verifier, showing that anchor placement matters in the tested settings. These comparisons do not evaluate Wang et al.'s calibrated multi-anchor rule, which the paper itself identifies as closely related. I would like to see a comparison with that rule at the same KV budget and comparable measured selection cost, accounting for both calibration and online measurement costs. This would clarify whether SKIVE improves the quality/throughput tradeoff or achieves a comparable tradeoff without offline calibration.
Questions for authors
Q1. Comparison is needed to assess the contribution.
- How does the online controller compare with Wang et al.'s calibrated multi-anchor rule at the same KV budget and comparable measured selection cost, including the calibration/setup cost? I am interested in whether the benefit is better throughput and quality, or comparable performance without offline calibration.
Q2. Clarification of the quality constraint.
- Table 3 chooses sparsity using ROUGE-L on the same GovReport documents that are reported, and Table 14 gives a 90% equivalence bound of approximately for the 64K online configuration without refresh. The appendix explicitly reports this limitation, and the arXiv transfer results provide positive evidence of transfer. Is there an evaluation in which sparsity was fixed independently of the evaluated documents, and does the uncertainty support the stated 5% tolerance? Otherwise, the quality floor should be described as an in-sample operating-point criterion, rather than evidence that degradation is bounded by 5%.
2 comments
Response to Reviewer WSLK (2/2): comparison with Wang et al., the quality constraint
W2 and Q1. The calibrated multi-anchor rule of Wang et al.
We implemented their rule: layer 0 plus the layers whose selection overlaps least with the previous layer's on calibration documents ("Jaccard anchors" below). We ran it at the same KV budget and the same number of anchors as ours (Table 14). Their paper leaves the number of anchors as a free parameter and gives none for LLaMA-3.1-8B, so we gave it ours.
The answer to your question: comparable quality and throughput, without offline calibration.
| Change in TP points against every-layer selection (s = 0.95, six anchors) | 16K | 32K | 64K |
|---|---|---|---|
| Jaccard anchors as published (scoring the root query only) | −2.2† | −0.7 | −0.1 |
| Jaccard anchors, tree nodes scored our way | −1.0* | −0.4 | −0.4 |
| Our solver's six anchors | −0.1 | −0.3 | −0.1 |
| Online controller (chooses 6–8 anchors itself) | −0.8 | −0.1 | 0.0 |
(* p < 0.05, † p < 0.01.) End to end at 64K with six anchors, the Jaccard anchors reach 5.92× AR at −3.3% ROUGE-L and our solver's anchors 5.96× at −3.4%.
What ours adds:
- No calibration. No calibration documents and no calibration pass.
- It chooses the number of anchors (Eq. 4). Their rule has to be given it.
- It keeps adapting while decoding. This costs about 3% of speed at 64K: 5.50× AR for the controller against 5.65× for anchors fixed offline, in the same run.
The two rules agree because they detect the same thing, the layers where a reused selection goes stale (Fig. 2a): theirs through set overlap, ours through retained mass. When two schedules differ by one TP point or less, neither rule is reliably ahead.
The row label. The row formerly labelled "Root reuse (Wang)" selects once at layer 0. It is now labelled "Root reuse (layer 0)", and §5.1 states which rows measure the two published variants: "Root query, every layer" (Table 2) and "Jaccard anchors" (Table 14).
Q2. The quality constraint.
The 5% in Table 3 is the threshold of the rule that picked each sparsity: the sparsest setting whose loss on those fifty documents is within 5%. You are right that this rule was applied to the same documents the table reports. The caption now calls it an in-sample criterion, and we evaluated the same sparsities on documents that played no part in choosing them (App. B.4, Table 12). ROUGE-L change against AR for the online controller:
| 4K | 16K | 32K | 64K | |
|---|---|---|---|---|
| Fifty held-out GovReport documents | −5.3% | −7.5%* | −3.5% | too few documents |
| Held-out and Table 3's documents together (100) | −5.0%* | −5.1%* | −4.1%* | |
| Fifty held-out arXiv documents | −0.8% | |||
| Sparsity chosen on arXiv, scored on GovReport | −5.6%* | −5.4%* | −4.7% | −4.8% |
- The loss at these sparsities is about 5% of AR's ROUGE-L, and the paper states it as that.
- The uncertainty. The 90% equivalence bound is ±5.9% on the held-out arXiv documents at 64K and ±7.5–8.4% on the hundred GovReport documents. So the data rule out a loss larger than 6–8%. They do not make 5% a guarantee, and Table 3 uses it only as the selection threshold.
- Where the loss comes from: the sparse cache, not our schedule. Every-layer selection loses 5.1–5.9% on the same hundred documents. At 16K, AR reading the same sparse cache, with no speculation at all, loses 7.3%. On the held-out documents the controller's ROUGE-L is within two standard errors of every-layer selection's (they differ by 1.2% or less; two standard errors is 3%), while the controller is 1.2–1.3× faster.
Presentation. §4.1 and App. B.1 are reorganized into what is assumed, what is tested and what the method needs. We would value your reading of them.
Response to Reviewer WSLK (1/2): the derivation
Thank you for the careful reading of the derivation. You are right on each point. The method and all results are unchanged; what changed is what the paper claims, and how it supports it.
1. The starting equality was an approximation. §4.1 and App. B.1 now start from the exact identity you wrote,
E[TP]=∑vPr(Dv)⋅Pr(Sv∣Dv)(Eq. 6)
where Dv and Sv mean that the dense and the sparse verifier accept node v. The paper then names the two substitutions: the drafter's prior stands in for the first factor, and the second factor is modelled as rising with the mass the selected blocks retain for v. Eq. 1 and Eq. 3 are now called surrogates, as you suggest, and the solver is described as exact for Eq. 3, not for expected TP.
2. The schedule-level comparison you asked for (App. B.1, Fig. 5). We ran fourteen schedules, each under its own selections, so the queries are recomputed under each schedule: our solver's anchors at six counts from 3 to 14, the calibrated anchors of Wang et al. at four counts, two hand-made schedules, five uniform anchors, and every-layer selection. Ranking them by Eq. 3 and by measured TP gives nearly the same order. The Spearman rank correlation is 0.81, 0.92, 0.88 and 0.90 in the four settings (16K and 32K, s = 0.90 and 0.95; all p < 0.001; 1 means identical order).
This test compares the objective with the final verification outcome, so it covers your points about the layer sum. It does not assume that TP decomposes over layers, and it runs every schedule with its own reused selections upstream. You are also right that equal layer weights do not follow from Eq. 1. §4.3 now states them as a design choice.
3. The per-node test of the linear step (App. B.1, Fig. 4). We measured acceptance against retained mass with the node held fixed: the same node, in the same round, at the same budget, under four different selections.