- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KAP, a system that translates structured frontend knowledge into runtime KV-cache access plans. Its GraphSpec implementation constructs selected KV views from full-context prefill while preserving logical token positions. The same model generates proposals using these views and verifies them against the full context under top1 and top3 acceptance threshold. The evaluation reports decoding gains under top3 verification and explores transfer across context lengths, models, retrieval frontends, and QA workloads.
Critical strengths
- The paper presents a clear abstraction that separates frontend evidence selection from backend KV access. It also has concrete systems implementation, with integration into real LLM serving engine.
- The experiments is comprehensive which covers multiple models, retrieval frontends, QA workloads, and context lengths.
Critical weaknesses
- Insufficient evidence for the benefit of graph-guided selection. In the primary setting, strict top1 GraphSpec is slower than full-context decoding at every evaluated context length, leaving its practical acceleration benefit unclear.
- Missing competitive speculative-decoding baselines. The evaluation omits comparisons with strong small-model or learned drafters, such as EAGLE and native MTP where supported, under greedy matched output.
- Insufficient validation of baseline reimplementations. The modified vLLM implementations of MagicDec-SnapKV and RAPID-core are substantially slower than full-context decoding, whereas their original papers report acceleration in different settings. Without calibration against the released implementations or profiling that explains this performance reversal, it is difficult to determine whether the reported advantages reflect methodological improvements or implementation differences.
- GraphSpec still relies on full context to verify. The contribution is limited if it neither saves space nor providing more speedup over traditional speculative decoding methods.
Questions for authors
See weaknesses.
4 comments
Dear Reviewer dENe,
Thank you again for your rigorous critique. We have also uploaded the revised PDF (Section 3, RQ5/Table 9, Appendix A.5, and Appendix C.4/Table 17) incorporating the greedy Llama-3.1-8B comparison against EAGLE-3 (3.17x–9.30x Top-1; 3.49x–10.73x Top-3 at 32K–128K), unmodified native RAPID (1.46x–1.83x) and MagicDec (1.07x–1.85x) calibrations, and verification-free DirectKV (>50% peak GPU memory savings at 128K and 8.8x TTFT / 2.74x E2E reduction at 32K).
As the author–reviewer discussion period closes on November 18, we would be grateful to know if any concerns remain before the discussion closes.
Thanks for the response.
-
The added experiments of EAGLE-3 has an extremely low acceptance rate, with a lower throughput than original vLLM decoding. I understand that might due to the mismatch between training set of
RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3and test set in KAP. The same for RAPID and MagicDec. In this case, it is essential to compare the Top-1 performance with vLLM full context decoding. And it is even better if the author can provide results on HumanEval and GSM8K. -
The good performance on static sparsity raises my further concern about whether this RAG+speculative-decoding pipeline is necessary. According to the Table1, there are only <1% useful evidence in the input context, so it is easy to understand dropping over 85% context still remains good performance.
KAP does full prefill pass to generate KV cache and access with selection during decoding, but in this kind of QA task, prefill dominants the time. The authors separate TTFT and TPS which makes the speedup looks significant, but the e2e improvement is little. For example, in Table3, the decoding throughput is 1.17 while e2e improvement is 2%, and this is on relaxed Top-3 policy, not Top-1. In contrast, static sparse KV selection, directKV(select), speeds up 8.8x TTFT with only 2% accuracy drop. This result put GraphSpec at an awkward position: For better efficiency, drop KV statically improves e2e time with little performance degeneration, but that attributes to the workload nature and RAG-retrival; For better accuracy, full context request completion time is already competitive.
Based on these, I would keep my scoring.
Thank you for the follow-up. We address the three concerns directly.
1. Strict Top-1 versus full-context vLLM. We place the paired full-context greedy measurements alongside the EAGLE-3 and GraphSpec results from Table 9A and our first response (G1). All rows use Llama-3.1-8B-Instruct, TP=2, greedy decoding, 100 questions per workload/length, and a 512-token output cap.
| Workload | Length | vLLM greedy | EAGLE-3 | GraphSpec Top-1 | Top-1/vLLM |
|---|---|---|---|---|---|
| SPIQA | 4K | 81.72 | 84.67 (29.4%) | 107.64 (85.1%) | 1.32x |
| SPIQA | 32K | 75.07 | 27.18 (3.7%) | 104.02 (83.2%) | 1.39x |
| SPIQA | 128K | 58.04 | 9.23 (1.8%) | 85.84 (79.4%) | 1.48x |
| MuSiQue | 4K | 83.13 | 154.51 (31.2%) | 111.10 (84.6%) | 1.34x |
| MuSiQue | 32K | 75.97 | 32.56 (3.5%) | 103.36 (82.8%) | 1.36x |
| MuSiQue | 128K | 67.34 | 9.92 (2.8%) | 87.01 (81.1%) | 1.29x |
Strict Top-1 achieves 1.29-1.48x the decode throughput of ordinary vLLM in these six Llama settings, showing frontend-guided selection accelerates decoding without relying on EAGLE-3 or relaxed Top-3.
SPIQA (scientific-document QA) and MuSiQue (multi-hop QA) were in the original submission: both supply long contexts with evidence selected upstream. We evaluated EAGLE-3 on those same workloads during review. EAGLE-3 beats greedy on both 4K workloads and is fastest on MuSiQue at 4K, before its acceptance drops from ~30% at 4K to 1.8%-3.7% at 32K-128K—isolating long-context draft-head degradation rather than dataset mismatch. RAPID and MagicDec-SnapKV can also be workload-sensitive, but the native configurations we test draft from retrieved text and attention-selected KV, respectively. Neither uses a separately trained draft head. At 32K, GraphSpec Top-1 achieves 1.46-1.59x RAPID's and 1.07-1.85x MagicDec-SnapKV's TPS on their respective native stacks (Table 9B/G3), ruling out checkpoint mismatch as the cause. HumanEval and GSM8K would test the learned head on code and math, but their standard inputs lack the retrieval priorities and long context KAP targets; we did not evaluate them and make no claim there.
2. Evidence sparsity is not a KV deletion rule. Table 1 measures the fraction of prompt tokens in passages matched to reference annotations, not the fraction useful to the model. At 4K the figures are 14.76% on SPIQA and 7.63% on MuSiQue; at 128K they are 0.73% and 0.27%. At inference the runtime receives frontend priorities, not gold evidence locations. Knowing the fraction does not tell it which passages it can safely drop.
Table 4 shows why this distinction matters on SPIQA at 128K. Random concat and Local window achieve 42.2% and 47.6% pass rates. Physical concat uses the Graph-RAG selection and reaches 76.5%; GraphSpec Top-3 reaches 84.3%, versus 84.6% for full context. These controls shorten the input. GraphSpec instead restricts proposal-side KV access, leaving even unselected passages available to the verifier. Thus both what is selected and where the selection is enforced matter.
3. Why GraphSpec is neither redundant with DirectKV nor superseded by full-context decoding. Table 3's 1.95% E2E gain reflects Amdahl's Law in single-turn short-output QA (~110 tokens), where 128K prefill takes 93.4% of latency. Whenever prefill is amortized (multi-turn/prefix-cached serving, longer outputs) or decode has a larger share, GraphSpec's decode speedups translate directly into E2E gains: on native RAPID at 32K (G3), Verified GraphSpec (Top-1/Top-3) cuts mean E2E from 9.48 to 7.02/6.67 s (SPIQA) and 6.63 to 5.57/5.50 s (MuSiQue); on native MagicDec at 32K (G3), GraphSpec Top-1 cuts mean E2E from 9.04 to 7.79 s (SPIQA) and 7.58 to 7.41 s (MuSiQue); and at c=2-8 (Table 17B), GraphSpec Top-3 sustains 1.01-1.18x batch TPS over vLLM—showing real E2E and serving gains beyond Table 3.
More fundamentally, DirectKV and GraphSpec are complementary Level-3 realizations of KAP's three-level long-context design: (1) Architecture—a unified Runtime Access Plan IR; (2) Mechanism—structure- and position-preserving physical KV access ( pos); and (3) Optimization (Eq. 2). Static dropping via raw RAG fails in Table 4 (Physical concat drops to 75.7% at 32K vs. 86.9%). DirectKV reaches 90% pass with full prefill and 88% with selective prefill on SPIQA 32K, yielding 8.8x TTFT (Table 17A) and 6.36-39.35x concurrent p50 E2E speedups over vLLM (Table 17C) specifically because it executes KAP's IR, not naive RAG dropping. When priors are clean and prefill/KV dominates, KAP lowers plans to DirectKV (Mode 1); when priors may omit bridge facts or full-context checks are needed, KAP lowers plans to GraphSpec (Mode 2)—delivering 1.29-1.48x Top-1 over vLLM, 1.46-1.83x TPS / up to 1.42x E2E on RAPID, and 1.07-1.85x TPS / 1.16x E2E on MagicDec. More broadly, because KAP is the core abstraction, the same IR lowers to additional backend realizations as needed.
We hope this direct comparison and clarification of the latency results inform your final assessment.
Response to Reviewer dENe: Graph-Guided Selection & Top-1, EAGLE-3 (3.17x–10.73x), Native RAPID/MagicDec Ports, and DirectKV
We thank Reviewer
dENefor the rigorous critique and address W1–W4 below (Global ResponseG1–G4).1. W1: Theory-to-Practice Top-1 & Graph Selection
Per Eq. (2) (KVcrit≈h−s(1−h)Wmain),
Qwen3-VL-32B's large weight-read cost Wmain and moderate Top-1 acceptance (h≈48%–53%) keep KVcrit high, whereas Top-3 (h≈95%) collapses (1−h)Wmain for 128K speedups (1.17x). OnLlama-3.1-8B-Instruct, smaller weight footprint Wmain and high Top-1 acceptance (h≈82%, 79.4%–85.1%) drop KVcrit sharply—enabling strict Top-1 speedups across all stacks (85.84–111.10 TPS 4K–128K vLLM, G1; 30.38–30.78 TPS / 1.46x–1.59x native RAPID; 65.66–66.84 TPS / 1.07x–1.85x native MagicDec, G3). Under the identical GraphSpec executor (16.0% vs 16.3%KV, Table 5/10),Graph-RAGvs.BGE-M3raises acceptance (91.9%->94.7%) and TPS (47.47->52.59), isolating graph-guided selection.2. W2: Learned Drafters (EAGLE-3)
Neither model ships native MTP heads. Under greedy decoding (
temp=0), we benchmarkedLlama-3.1-8B-Instruct(vLLM v0.16, TP=2, 2x A800, max=512, 100 random QAs/bucket, 1,800 runs) vs. EAGLE-3 (RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3):EAGLE-3 collapses from ~30% acceptance (4K) to 1.8%–3.7% (32K–128K); GraphSpec Top-1 sustains 79.4%–85.1% without any learned draft head, beating EAGLE-3 by 3.17x–9.30x (Top-1) and 3.49x–10.73x (Top-3) at 32K–128K.
3. W3: Unmodified Official RAPID & MagicDec Calibration
To rule out vLLM porting bias, we kept official RAPID & MagicDec intact (SHA-256/AST verified) and ported KAP (
GraphSpec/DirectKV) onto their native stacks (Llama-3.1-8B-Instruct, 1x A800, 32K, 100 random QAs/dataset, identical frozen selections15.0%/8.8%to control variables, G3):22d41f4, probabilistic):LongSpecEnginehits 19.41 / 20.85 TPS (SPIQA/MuSiQue, matching RAPID Table 2 ~18.18 TPS). GraphSpec hits 30.78 / 30.38 TPS (1.59x / 1.46x, Top-1) and 35.59 / 35.65 TPS (1.83x / 1.71x, Top-3); DirectKV hits 66.56–68.45 TPS (3.27x–3.45x) and 1.53s–2.31s QA time.09cd671, strict Top-1):LMBackendSnapKV hits 36.21 / 61.41 TPS (SPIQA/MuSiQue). Under strict Top-1, GraphSpec hits 66.84 / 65.66 TPS (1.85x / 1.07x); DirectKV hits 80.50–81.71 TPS (1.33x–2.25x) and 2.52s–2.76s QA time. SnapKV's static γ=7 (64%–73% acceptance) fails to amortize verification here, whereas GraphSpec achieves higher frontend-guided acceptance (80%–83%Top-1) with dynamic K and compact workspaces.4. W4: Verification-Free Memory Savings & Flexibility (
DirectKV)Even with full verification, GraphSpec beats EAGLE-3 (3.17x–10.73x), RAPID (1.46x–1.83x), and MagicDec-SnapKV (1.07x–1.85x). KAP's contract also lowers the same plan into verification-free
DirectKV(ρverify=none) preserving originalpos(G2):DirectKV (full_prefill)runs page-aligned in-place KV compaction and frees unselected KV post-prefill, matching Full-Context SPIQA 32K pass rate onQwen3-VL-32B(90.0% vs 90.0%, 56.18 TPS, s=15.3%).DirectKV (selective_prefill)compiles priors into a position-preserving sparse prefill+decode schedule retaining 15.3% (32K) / 7.2% (128K) prompt KV—hitting 88.0% pass (vs 90.0%), 59.52 TPS, 8.8x TTFT (4.48s->0.51s), and 2.74x E2E (6.79s->2.48s) onQwen3-VL-32BSPIQA 32K, and >50% peak GPU memory savings at 128K on native SPIQA ports.Thus, KAP flexibly compiles one access contract into multiple realizations: Verified GraphSpec enforces full-context verification at every step to guarantee answer fidelity against frontend omissions (Table 4/8, G5), while DirectKV trades verification for maximum memory and latency savings when frontend precision is high. By adapting these realizations to frontend retrieval confidence and backend serving constraints (memory, latency, or fidelity requirements), KAP adjusts seamlessly across deployment regimes. We respectfully ask if you might reconsider your evaluation.