acceptodds
Under review as a conference paper at ICLR 2027

KAP: Bridging the Knowledge Selection–Runtime Consumption Gap in LLM Systems

Abstract

Modern LLM systems increasingly rely on knowledge-selection pipelines that produce structured signals, such as evidence rankings, graph relations, multimodal alignment, and confidence scores. However, these signals are typically discarded when knowledge is serialized into a flat prompt, leaving the serving system unaware of which knowledge is important during decoding. This creates a gap between knowledge selection and runtime consumption: the model is presented with a complete logical context, while the runtime lacks the information needed to avoid unnecessary KV access. We propose Knowledge Access Planning (KAP), an execution abstraction that bridges this gap by carrying structured knowledge priors into LLM serving through a universal intermediate representation (IR), the runtime access plan. KAP separates logical prompt semantics from physical execution: its compiler translates knowledge-selection priorities into backend-executable plans, while its executor selectively accesses physical KV states according to these plans, without modifying the model, prompt semantics, or training procedure. We instantiate KAP with GraphSpec, a reference realization that executes runtime access plans within vLLM, and derive a conditional cost model characterizing its positive-speedup regime. Across a range of experimental settings, GraphSpec reduces proposal-path KV exposure to 5.1–7.9% of the source state at 128K and translates this reduction into long-context decode speedups while maintaining answer quality comparable to full-context decoding. These results demonstrate KAP's effectiveness in substantially reducing proposal-path KV exposure. More broadly, they show that knowledge-selection signals can be compiled into actionable execution plans, extending their role beyond prompt construction to efficient LLM decoding.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
efficient LLM inference, long-context inference, KV-cache optimization, retrieval-augmented generation, speculative decoding

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_dENeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KAP, a system that translates structured frontend knowledge into runtime KV-cache access plans. Its GraphSpec implementation constructs selected KV views from full-context prefill while preserving logical token positions. The same model generates proposals using these views and verifies them against the full context under top1 and top3 acceptance threshold. The evaluation reports decoding gains under top3 verification and explores transfer across context lengths, models, retrieval frontends, and QA workloads.

Critical strengths
  1. The paper presents a clear abstraction that separates frontend evidence selection from backend KV access. It also has concrete systems implementation, with integration into real LLM serving engine.
  2. The experiments is comprehensive which covers multiple models, retrieval frontends, QA workloads, and context lengths.
Critical weaknesses
  1. Insufficient evidence for the benefit of graph-guided selection. In the primary setting, strict top1 GraphSpec is slower than full-context decoding at every evaluated context length, leaving its practical acceleration benefit unclear.
  2. Missing competitive speculative-decoding baselines. The evaluation omits comparisons with strong small-model or learned drafters, such as EAGLE and native MTP where supported, under greedy matched output.
  3. Insufficient validation of baseline reimplementations. The modified vLLM implementations of MagicDec-SnapKV and RAPID-core are substantially slower than full-context decoding, whereas their original papers report acceleration in different settings. Without calibration against the released implementations or profiling that explains this performance reversal, it is difficult to determine whether the reported advantages reflect methodological improvements or implementation differences.
  4. GraphSpec still relies on full context to verify. The contribution is limited if it neither saves space nor providing more speedup over traditional speculative decoding methods.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dENe: Graph-Guided Selection & Top-1, EAGLE-3 (3.17x–10.73x), Native RAPID/MagicDec Ports, and DirectKV

We thank Reviewer dENe for the rigorous critique and address W1–W4 below (Global Response G1–G4).

1. W1: Theory-to-Practice Top-1 & Graph Selection

Per Eq. (2) (KVcrit≈(1−h)Wmainh−sKV_{\text{crit}}\approx\frac{(1-h)W_{\text{main}}}{h-s}), Qwen3-VL-32B's large weight-read cost WmainW_{\text{main}} and moderate Top-1 acceptance (h≈48%–53%h\approx 48\%\text{--}53\%) keep KVcritKV_{\text{crit}} high, whereas Top-3 (h≈95%h\approx 95\%) collapses (1−h)Wmain(1-h)W_{\text{main}} for 128K speedups (1.17x). On Llama-3.1-8B-Instruct, smaller weight footprint WmainW_{\text{main}} and high Top-1 acceptance (h≈82%h\approx 82\%, 79.4%–85.1%) drop KVcritKV_{\text{crit}} sharply—enabling strict Top-1 speedups across all stacks (85.84–111.10 TPS 4K–128K vLLM, G1; 30.38–30.78 TPS / 1.46x–1.59x native RAPID; 65.66–66.84 TPS / 1.07x–1.85x native MagicDec, G3). Under the identical GraphSpec executor (16.0% vs 16.3% KV, Table 5/10), Graph-RAG vs. BGE-M3 raises acceptance (91.9%->94.7%) and TPS (47.47->52.59), isolating graph-guided selection.

2. W2: Learned Drafters (EAGLE-3)

Neither model ships native MTP heads. Under greedy decoding (temp=0), we benchmarked Llama-3.1-8B-Instruct (vLLM v0.16, TP=2, 2x A800, max=512, 100 random QAs/bucket, 1,800 runs) vs. EAGLE-3 (RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3):

BucketEAGLE-3 TPS (h)GS Top-1 TPS (h / vs E3)GS Top-3 TPS (h / vs E3)
SPIQA 4K84.67 (29.4%)107.64 (85.1% / 1.27x)124.34 (96.8% / 1.47x)
SPIQA 32K27.18 (3.7%)104.02 (83.2% / 3.83x)116.74 (96.3% / 4.29x)
SPIQA 128K9.23 (1.8%)85.84 (79.4% / 9.30x)99.07 (95.1% / 10.73x)
MuSiQue 4K154.51 (31.2%)111.10 (84.6% / 0.72x)122.58 (96.5% / 0.79x)
MuSiQue 32K32.56 (3.5%)103.36 (82.8% / 3.17x)113.70 (96.1% / 3.49x)
MuSiQue 128K9.92 (2.8%)87.01 (81.1% / 8.77x)92.37 (95.4% / 9.31x)

EAGLE-3 collapses from ~30% acceptance (4K) to 1.8%–3.7% (32K–128K); GraphSpec Top-1 sustains 79.4%–85.1% without any learned draft head, beating EAGLE-3 by 3.17x–9.30x (Top-1) and 3.49x–10.73x (Top-3) at 32K–128K.

3. W3: Unmodified Official RAPID & MagicDec Calibration

To rule out vLLM porting bias, we kept official RAPID & MagicDec intact (SHA-256/AST verified) and ported KAP (GraphSpec/DirectKV) onto their native stacks (Llama-3.1-8B-Instruct, 1x A800, 32K, 100 random QAs/dataset, identical frozen selections 15.0%/8.8% to control variables, G3):

  • Official RAPID (HF+FA2, 22d41f4, probabilistic): LongSpecEngine hits 19.41 / 20.85 TPS (SPIQA/MuSiQue, matching RAPID Table 2 ~18.18 TPS). GraphSpec hits 30.78 / 30.38 TPS (1.59x / 1.46x, Top-1) and 35.59 / 35.65 TPS (1.83x / 1.71x, Top-3); DirectKV hits 66.56–68.45 TPS (3.27x–3.45x) and 1.53s–2.31s QA time.
  • Official MagicDec (FlashInfer, 09cd671, strict Top-1): LMBackend SnapKV hits 36.21 / 61.41 TPS (SPIQA/MuSiQue). Under strict Top-1, GraphSpec hits 66.84 / 65.66 TPS (1.85x / 1.07x); DirectKV hits 80.50–81.71 TPS (1.33x–2.25x) and 2.52s–2.76s QA time. SnapKV's static γ=7\gamma=7 (64%–73% acceptance) fails to amortize verification here, whereas GraphSpec achieves higher frontend-guided acceptance (80%–83% Top-1) with dynamic KK and compact workspaces.

4. W4: Verification-Free Memory Savings & Flexibility (DirectKV)

Even with full verification, GraphSpec beats EAGLE-3 (3.17x–10.73x), RAPID (1.46x–1.83x), and MagicDec-SnapKV (1.07x–1.85x). KAP's contract also lowers the same plan into verification-free DirectKV (ρverify=none\rho_{\text{verify}}=\text{none}) preserving original pos (G2):

  • DirectKV (full_prefill) runs page-aligned in-place KV compaction and frees unselected KV post-prefill, matching Full-Context SPIQA 32K pass rate on Qwen3-VL-32B (90.0% vs 90.0%, 56.18 TPS, s=15.3%s=15.3\%).
  • DirectKV (selective_prefill) compiles priors into a position-preserving sparse prefill+decode schedule retaining 15.3% (32K) / 7.2% (128K) prompt KV—hitting 88.0% pass (vs 90.0%), 59.52 TPS, 8.8x TTFT (4.48s->0.51s), and 2.74x E2E (6.79s->2.48s) on Qwen3-VL-32B SPIQA 32K, and >50% peak GPU memory savings at 128K on native SPIQA ports.

Thus, KAP flexibly compiles one access contract into multiple realizations: Verified GraphSpec enforces full-context verification at every step to guarantee answer fidelity against frontend omissions (Table 4/8, G5), while DirectKV trades verification for maximum memory and latency savings when frontend precision is high. By adapting these realizations to frontend retrieval confidence and backend serving constraints (memory, latency, or fidelity requirements), KAP adjusts seamlessly across deployment regimes. We respectfully ask if you might reconsider your evaluation.

Dear Reviewer dENe,

Thank you again for your rigorous critique. We have also uploaded the revised PDF (Section 3, RQ5/Table 9, Appendix A.5, and Appendix C.4/Table 17) incorporating the greedy Llama-3.1-8B comparison against EAGLE-3 (3.17x–9.30x Top-1; 3.49x–10.73x Top-3 at 32K–128K), unmodified native RAPID (1.46x–1.83x) and MagicDec (1.07x–1.85x) calibrations, and verification-free DirectKV (>50% peak GPU memory savings at 128K and 8.8x TTFT / 2.74x E2E reduction at 32K).

As the author–reviewer discussion period closes on November 18, we would be grateful to know if any concerns remain before the discussion closes.

Thanks for the response.

  1. The added experiments of EAGLE-3 has an extremely low acceptance rate, with a lower throughput than original vLLM decoding. I understand that might due to the mismatch between training set of RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3 and test set in KAP. The same for RAPID and MagicDec. In this case, it is essential to compare the Top-1 performance with vLLM full context decoding. And it is even better if the author can provide results on HumanEval and GSM8K.

  2. The good performance on static sparsity raises my further concern about whether this RAG+speculative-decoding pipeline is necessary. According to the Table1, there are only <1% useful evidence in the input context, so it is easy to understand dropping over 85% context still remains good performance.

KAP does full prefill pass to generate KV cache and access with selection during decoding, but in this kind of QA task, prefill dominants the time. The authors separate TTFT and TPS which makes the speedup looks significant, but the e2e improvement is little. For example, in Table3, the decoding throughput is 1.17 while e2e improvement is 2%, and this is on relaxed Top-3 policy, not Top-1. In contrast, static sparse KV selection, directKV(select), speeds up 8.8x TTFT with only 2% accuracy drop. This result put GraphSpec at an awkward position: For better efficiency, drop KV statically improves e2e time with little performance degeneration, but that attributes to the workload nature and RAG-retrival; For better accuracy, full context request completion time is already competitive.

Based on these, I would keep my scoring.

Thank you for the follow-up. We address the three concerns directly.

1. Strict Top-1 versus full-context vLLM. We place the paired full-context greedy measurements alongside the EAGLE-3 and GraphSpec results from Table 9A and our first response (G1). All rows use Llama-3.1-8B-Instruct, TP=2, greedy decoding, 100 questions per workload/length, and a 512-token output cap.

WorkloadLengthvLLM greedyEAGLE-3GraphSpec Top-1Top-1/vLLM
SPIQA4K81.7284.67 (29.4%)107.64 (85.1%)1.32x
SPIQA32K75.0727.18 (3.7%)104.02 (83.2%)1.39x
SPIQA128K58.049.23 (1.8%)85.84 (79.4%)1.48x
MuSiQue4K83.13154.51 (31.2%)111.10 (84.6%)1.34x
MuSiQue32K75.9732.56 (3.5%)103.36 (82.8%)1.36x
MuSiQue128K67.349.92 (2.8%)87.01 (81.1%)1.29x

Strict Top-1 achieves 1.29-1.48x the decode throughput of ordinary vLLM in these six Llama settings, showing frontend-guided selection accelerates decoding without relying on EAGLE-3 or relaxed Top-3.

SPIQA (scientific-document QA) and MuSiQue (multi-hop QA) were in the original submission: both supply long contexts with evidence selected upstream. We evaluated EAGLE-3 on those same workloads during review. EAGLE-3 beats greedy on both 4K workloads and is fastest on MuSiQue at 4K, before its acceptance drops from ~30% at 4K to 1.8%-3.7% at 32K-128K—isolating long-context draft-head degradation rather than dataset mismatch. RAPID and MagicDec-SnapKV can also be workload-sensitive, but the native configurations we test draft from retrieved text and attention-selected KV, respectively. Neither uses a separately trained draft head. At 32K, GraphSpec Top-1 achieves 1.46-1.59x RAPID's and 1.07-1.85x MagicDec-SnapKV's TPS on their respective native stacks (Table 9B/G3), ruling out checkpoint mismatch as the cause. HumanEval and GSM8K would test the learned head on code and math, but their standard inputs lack the retrieval priorities and long context KAP targets; we did not evaluate them and make no claim there.

2. Evidence sparsity is not a KV deletion rule. Table 1 measures the fraction of prompt tokens in passages matched to reference annotations, not the fraction useful to the model. At 4K the figures are 14.76% on SPIQA and 7.63% on MuSiQue; at 128K they are 0.73% and 0.27%. At inference the runtime receives frontend priorities, not gold evidence locations. Knowing the fraction does not tell it which passages it can safely drop.

Table 4 shows why this distinction matters on SPIQA at 128K. Random concat and Local window achieve 42.2% and 47.6% pass rates. Physical concat uses the Graph-RAG selection and reaches 76.5%; GraphSpec Top-3 reaches 84.3%, versus 84.6% for full context. These controls shorten the input. GraphSpec instead restricts proposal-side KV access, leaving even unselected passages available to the verifier. Thus both what is selected and where the selection is enforced matter.

3. Why GraphSpec is neither redundant with DirectKV nor superseded by full-context decoding. Table 3's 1.95% E2E gain reflects Amdahl's Law in single-turn short-output QA (~110 tokens), where 128K prefill takes 93.4% of latency. Whenever prefill is amortized (multi-turn/prefix-cached serving, longer outputs) or decode has a larger share, GraphSpec's decode speedups translate directly into E2E gains: on native RAPID at 32K (G3), Verified GraphSpec (Top-1/Top-3) cuts mean E2E from 9.48 to 7.02/6.67 s (SPIQA) and 6.63 to 5.57/5.50 s (MuSiQue); on native MagicDec at 32K (G3), GraphSpec Top-1 cuts mean E2E from 9.04 to 7.79 s (SPIQA) and 7.58 to 7.41 s (MuSiQue); and at c=2-8 (Table 17B), GraphSpec Top-3 sustains 1.01-1.18x batch TPS over vLLM—showing real E2E and serving gains beyond Table 3.

More fundamentally, DirectKV and GraphSpec are complementary Level-3 realizations of KAP's three-level long-context design: (1) Architecture—a unified Runtime Access Plan IR; (2) Mechanism—structure- and position-preserving physical KV access (U,M,U, M, pos); and (3) Optimization (Eq. 2). Static dropping via raw RAG fails in Table 4 (Physical concat drops to 75.7% at 32K vs. 86.9%). DirectKV reaches 90% pass with full prefill and 88% with selective prefill on SPIQA 32K, yielding 8.8x TTFT (Table 17A) and 6.36-39.35x concurrent p50 E2E speedups over vLLM (Table 17C) specifically because it executes KAP's IR, not naive RAG dropping. When priors are clean and prefill/KV dominates, KAP lowers plans to DirectKV (Mode 1); when priors may omit bridge facts or full-context checks are needed, KAP lowers plans to GraphSpec (Mode 2)—delivering 1.29-1.48x Top-1 over vLLM, 1.46-1.83x TPS / up to 1.42x E2E on RAPID, and 1.07-1.85x TPS / 1.16x E2E on MagicDec. More broadly, because KAP is the core abstraction, the same IR lowers to additional backend realizations as needed.

We hope this direct comparison and clarification of the latency results inform your final assessment.

Reviewer_UKSvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Knowledge Access Planning (KAP), a framework that connects knowledge-selection signals from retrieval systems with the runtime execution of LLMs. KAP introduces a runtime access plan that allows the serving backend to selectively access KV cache states based on knowledge priorities. The authors implement GraphSpec within vLLM, combining selected-KV token proposals with full-context verification. Experiments on SPIQA and MuSiQue demonstrate reduced proposal-side KV access and improved decoding throughput while generally maintaining comparable answer quality.

Critical strengths
  1. The paper addresses an interesting gap between knowledge retrieval and LLM runtime execution. Leveraging frontend knowledge priorities to improve inference efficiency is a practically relevant direction.

  2. The proposed compiler–executor design provides a structured interface between knowledge-selection frontends and the serving backend, allowing retrieval information to influence runtime KV access.

  3. GraphSpec is integrated into vLLM without requiring changes to model parameters or training procedures. The consideration of KV page mapping, positional consistency, and runtime execution overhead strengthens the practical contribution.

  4. The paper includes comparisons with several related approaches, experiments across different context lengths, and transfer evaluations using different models, frontends, and workloads.

Critical weaknesses
  1. Although KAP is presented as a universal framework, the current implementation and evaluation are limited to the vLLM serving backend. Further discussion or validation across different serving architectures would strengthen the generality of the proposed runtime access plan.

  2. The main decoding speedups rely on Top-3 verification, which does not guarantee identical outputs to full-context decoding. The Top-1 configuration is substantially slower in some settings. This raises questions about the efficiency–quality trade-off when exact output equivalence is required.

  3. While the reduction in proposal-side KV access is substantial, the improvement in end-to-end latency is relatively modest. Further evaluation of practical deployment scenarios, including longer generations or concurrent requests, would help clarify the real-world benefits.

  4. The experiments use progressively extended contexts with additional hard negatives. Although this provides a useful controlled evaluation, experiments with more naturally occurring long-context workloads and varying knowledge-selection quality would further strengthen the generalizability of the results.

Questions for authors
  1. KAP is presented as a universal runtime abstraction, while the current evaluation focuses on vLLM. Could the authors further clarify which components of the runtime access plan are backend-independent and what challenges may arise when extending KAP to other serving frameworks?

  2. The reported decoding speedups mainly rely on Top-3 verification, whereas Top-1 verification appears less efficient in several settings. Could the authors further discuss the trade-off between exact output equivalence and decoding efficiency, particularly for applications requiring strict output consistency?

  3. Although GraphSpec substantially reduces proposal-side KV access, the reported improvements in end-to-end latency are relatively modest. Could the authors clarify which practical workload characteristics, such as generation length or request concurrency, are expected to provide the greatest end-to-end benefits?

  4. The experiments use controlled long-context inputs constructed by progressively adding hard negatives. Could the authors discuss how KAP would behave under more realistic workloads where the knowledge-selection frontend may provide incomplete or noisy evidence priorities?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UKSv: Cross-Backend Ports (RAPID & MagicDec), Top-1/Top-3/DirectKV, E2E Regimes, and Noisy Priors

We thank Reviewer UKSv for recognizing our problem formulation and engineering contributions. Please see Global Response (G1–G5) for full tables.

1. Extending KAP Beyond vLLM (W1, Q1)

We implemented and evaluated KAP on two additional serving architectures (G3): HF Transformers + FlashAttention-2 (native RAPID) using contiguous AppendCache buffers, and FlashInfer Paged KV Cache (native MagicDec) using paged indices.

  • Backend-Independent Contract (U,pos,ρaccess,ρverify)(U, pos, \rho_{\text{access}}, \rho_{\text{verify}}): Frontend outputs UU, preserved logical positions pos, and policies are 100% shared and unchanged across all 3 backends.
  • Backend-Specific Lowering (MM) & Challenges: vLLM compiles MM into request-leased PagedAttention arenas, coalesced cross-layer KV-sync kernels, and query-shape-dispatched (q=1 compact draft vs. q=k+1 verify) CUDA-graph replay with persistent VLM embedding contracts; HF-RAPID lowers MM to in-place AppendCache gathers and length-pointer rollbacks; FlashInfer-MagicDec lowers MM to paged-KV indices keyed by actual compact page geometry (last_page_len) to avoid cold JIT. Across both new backends, Verified GraphSpec outperforms native baselines (1.46x–1.83x vs. RAPID; 1.07x–1.85x vs. MagicDec-SnapKV) and DirectKV reaches 3.27x–3.45x (RAPID) and 1.33x–2.25x (MagicDec), supporting our framing of KAP as a Cross-Layer Runtime Access Contract (G5(3)).

2. Consistency vs. Efficiency Spectrum (W2, Q2)

Our cost boundary KVcrit≈(1−h)Wmainh−sKV_{\text{crit}} \approx \frac{(1-h)W_{\text{main}}}{h-s} (Eq. 2) explains how model weight cost WmainW_{\text{main}}, acceptance hh, and retained KV ratio ss govern the three operating points of ρverify\rho_{\text{verify}}:

  1. Strict Top-1 (exact greedy decision rule per step): On heavy 32B models (Qwen3-VL-32B), large per-step weight reads WmainW_{\text{main}} and moderate Top-1 acceptance (h≈48%–53%h\approx 48\%\text{--}53\%) keep KVcritKV_{\text{crit}} high. On Llama-3.1-8B-Instruct, smaller WmainW_{\text{main}} and high Top-1 acceptance (79.4%–85.1%) drop KVcritKV_{\text{crit}} sharply—allowing strict Top-1 to sustain 85.84–111.10 TPS across 4K–128K (3.17x–9.30x vs. EAGLE-3 at 32K–128K, G1) and beat Official RAPID (1.46x–1.59x) and MagicDec-SnapKV (1.07x–1.85x, G3).
  2. Tolerant Top-3: Raising hh to 94%–97% drives (1−h)Wmain→0(1-h)W_{\text{main}}\to 0 and collapses KVcritKV_{\text{crit}} even on 32B models—unlocking 1.17x speedup on Qwen3-VL-32B 128K and 99.07 TPS (10.73x vs. EAGLE-3) on Llama-3.1-8B 128K while keeping paired quality within [-6%,+4%] (128K) / [0%,+9%] (32K) (G5).
  3. Verification-Free DirectKV (ρverify=none\rho_{\text{verify}}=\text{none}): Eliminates the verification weight-read penalty and target decode KV residency altogether when frontend precision is high, delivering up to 1.87x concurrent speedup (128K c=4c=4) and 2.74x E2E reduction (selective_prefill 32K).

3. Where End-to-End Gains Are Largest (W3, Q3)

In single-turn short-output QA (~60–110 tokens), full prefill (TTFT) is 65%–92% of E2E time (e.g., 37.9s of 40.2s at 128K), bounding decode-only E2E gains by Amdahl's Law. New experiments highlight three high-gain regimes:

  1. Selective Prefill (DirectKV selective_prefill, G2): Cuts Qwen 32K TTFT 8.8x (4.48s -> 0.51s) and E2E 2.74x (6.79s -> 2.48s) with 88% vs. 90% pass rate; cuts Llama 32K QA time up to 4.1x (1.53s–2.76s).
  2. Concurrent Serving under Memory Limits (G4): DirectKV (selective_prefill) cuts concurrent KV-pool allocation by 75.3%–86.6% (32K c=8c=8) and 78.9%–82.9% (128K c=4c=4), boosts throughput by 1.46x–1.87x, and cuts p50 latency from 34.3–34.6s -> 3.2–5.6s (32K c=8c=8) and 147.5–148.1s -> 3.8–5.7s (128K c=4c=4).
  3. Multi-Turn / Prefix-Cached & Long Generations: Amortizes prefill so decode speedups (1.17x–1.87x) translate directly to wall-clock savings.

4. Natural Long Contexts & Noisy Priors (W4, Q4)

  • Natural Lengths: Before any sweep construction, native SPIQA and MuSiQue prompts already have medians of 59.7K tokens (rref=1.29%r_{\text{ref}}=1.29\%) and 26.3K tokens (rref=1.14%r_{\text{ref}}=1.14\%) (Table 1A).
  • Noisy/Incomplete Priors: Under degraded priors (BGE-M3 vs. Graph-RAG, Table 10), acceptance dips slightly (94.5%–95.4% -> 89.7%–94.6%) and TPS moderates (48.09 -> 43.50), but verification protects quality (86.8% vs. 84.3%), whereas unverified Random/Local compression collapses (42.2%–47.6%, Table 4). Dynamic KK adaptation also scales K→1K\to 1 on consecutive rejections to bound overhead.

Dear Reviewer UKSv,

Thank you again for your thoughtful review. We have also uploaded the revised PDF (Section 2, Section 3, RQ5/Table 9, and Appendix C.4/Table 17) incorporating the cross-backend ports onto unmodified native RAPID (HF+FA2) and MagicDec (FlashInfer), the Top-1 / Top-3 / DirectKV efficiency–consistency spectrum, and concurrent serving gains (75.3%–89.0% KV-pool savings and 6.4x–39.4x p50 latency reduction vs. official vLLM).

As the author–reviewer discussion period closes on November 18, please let us know if any remaining questions need further clarification.

Dear Reviewer UKSv,

In follow-up to your W2/Q2 on strict Top-1 efficiency, we would like to add the paired full-context greedy baseline corresponding to our Table 9A / Global Response G1 results on Llama-3.1-8B-Instruct (vLLM v0.16.0, TP=2, greedy, 100 QAs/bucket, max_tokens=512): strict Top-1 GraphSpec outperforms Official vLLM Full-Context Greedy by 1.29x–1.48x across all six 4K–128K SPIQA and MuSiQue buckets (107.64 / 104.02 / 85.84 vs. 81.72 / 75.07 / 58.04 TPS on SPIQA; 111.10 / 103.36 / 87.01 vs. 83.13 / 75.97 / 67.34 TPS on MuSiQue), without relying on Top-3 relaxation.

Thank you again for your time and thoughtful review.

Reviewer_EVfmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes Knowledge Access Planning (KAP), a compiler–executor abstraction that translates frontend knowledge-selection signals into explicit runtime access plans. These plans map selected evidence to backend KV objects while preserving original logical positions and specifying access and verification policies. Its vLLM implementation, GraphSpec, reuses KV states from full-context prefill for selective proposal generation, followed by full-context verification using the same model. Experiments across two QA datasets and two model families demonstrate decoding gains with tolerant Top-3 verification and broadly comparable answer quality.

Critical strengths
  1. Useful architecture with a concrete implementation. KAP explicitly connects frontend knowledge selection to backend KV access, and GraphSpec demonstrates this design through position-preserving mappings and integration with vLLM.

  2. Evaluation across relevant configurations. The paper tests different workloads, models, frontends, and verification policies, reporting answer quality and latency alongside throughput and explicitly acknowledging that Top-3 is not lossless.

Critical weaknesses
  1. Insufficient attribution of performance gains. Comparisons with reimplemented baselines do not clearly separate the benefits of knowledge-guided selection from runtime optimizations. Shared-executor comparisons at matched budgets are needed.

  2. Limited validation of the quality–efficiency trade-off. GraphSpec Top-1 is slower than full-context decoding throughout the primary SPIQA sweep, while acceleration relies on tolerant Top-3 verification. Similar aggregate quality scores, without paired uncertainty estimates, do not establish quality preservation.

Questions for authors
  1. Do GraphSpec and the reimplemented baselines share the same execution optimizations? Can you isolate selection effects using a shared executor and matched budgets, with a component-level timing breakdown?

  2. Under identical greedy generation settings, does GraphSpec Top-1 reproduce full-context token sequences? Please explain the reported pass-rate differences and provide paired uncertainty estimates for the quality changes introduced by Top-3.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EVfm: Isolating Selection vs. Executor Effects, Native Ports, Timing Breakdown, and Paired Bootstrap CIs

We thank Reviewer EVfm for the rigorous questions on performance attribution and statistical validation (see Global Response G1–G5 for full tables).

1. Isolating Selection Gains from Execution Optimizations (W1, Q1)

We separate knowledge-selection effects from executor engineering at three levels:

  1. Shared vLLM Controls (Table 4): GraphSpec, MagicDec-SnapKV, and RAPID-core all execute inside the same vLLM v0.16.0 engine (Qwen3-VL-32B, FlashAttention, CUDA-graph replay, k=6k=6, matched verification Top-1 for RAPID-core / Top-3 for MagicDec-SnapKV, and matched KV budgets ss). Their differences reflect inherent algorithmic KV representations (RAPID-core re-encodes selected text into a separate draft cache; MagicDec-SnapKV scores attention salience over full prefill), not unshared engine optimizations.
  2. 100% Identical-Executor Selection Ablation (Table 5 & Table 10): Under the exact same GraphSpec executor (identical CUDA-graph replay, workspace bucketing, and matched KV budget 16.0% vs. 16.3% at 32K), swapping only the frontend prior (Graph-RAG vs. BGE-M3) improves acceptance (91.9% -> 94.7% at 32K) and decode TPS (47.47 -> 52.59 at 32K; 43.50 -> 48.09 at 128K). With zero executor differences, this gain isolates pure knowledge-guided selection.
  3. Unmodified Official Native Ports (G3): Keeping official RAPID (HF+FA2) and MagicDec (FlashInfer) 100% unmodified (SHA-256/AST verified) on Llama-3.1-8B (32K, 100 random QAs with identical frozen selections to control variables; Top-1 on MagicDec to match its strict Top-1 rule, Top-1/3 on RAPID):
    • Native RAPID: Official RAPID achieves 19.41 / 20.85 TPS (SPIQA/MuSiQue); GraphSpec achieves 30.78 / 30.38 TPS (1.59x / 1.46x, Top-1) and 35.59 / 35.65 TPS (1.83x / 1.71x, Top-3).
    • Native MagicDec: Official SnapKV achieves 36.21 / 61.41 TPS; GraphSpec Top-1 achieves 66.84 / 65.66 TPS (1.85x / 1.07x).

Component-Level Timing Breakdown:

  • Plan Compilation & KV Materialization: 0.05s–0.15s (~1% of TTFT) in vLLM Table 3; 6–7 ms/req (<0.2% of request time) on native RAPID; 0.076 ms/req avg on native MagicDec after warmup.
  • Full-Context Prefill: Dominates TTFT (~3.5s–3.8s on Llama 32K RAPID; 16.65s / 18.19s on Llama 128K vLLM); DirectKV (selective_prefill) cuts 32K Qwen TTFT from 4.48s to 0.51s (8.8x).
  • Decode Loop: GraphSpec cuts Qwen 128K TPOT by -15.1% (24.64 -> 20.91 ms), beats EAGLE-3 by 9.30x–10.73x on Llama 128K (G1), and cuts 100-QA native RAPID decode time from 545.0s to 341.6s (Top-1) / 314.8s (Top-3).

2. Top-1 Equivalence, Pass-Rate Differences & Paired Uncertainty (W2, Q2)

(a) Why Top-1 and Top-3 pass rates differ slightly:

  • Top-1 vs. Full-Context: Algebraically, Top-1 accepts iff a draft matches the full-context argmax (identical greedy decision rule). In GPU execution, however, verification evaluates q=k+1q=k+1 queries whereas standard greedy evaluates q=1q=1 steps; non-associative BF16 reductions in FlashAttention/GEMM across query shapes can flip near-tie logits (e.g., "mention" vs. "describe" at token 6), shifting the subsequent prefix on a few prompts. Answer evals confirm Top-1 preserves quality (84% vs 86% native RAPID, 81% vs 84% native MagicDec, within ~1–2% on Qwen 128K). We updated Section 3 & Appendix A.3 to distinguish decision-rule consistency from bitwise identity.
  • Top-3 vs. Full-Context: Top-3 trades strict argmax matching for tolerant Top-3 membership (95%–97% acceptance), causing small symmetric per-QA flips (+gained / -lost).

(b) Paired Uncertainty Estimates (20,000-Replicate QA-Paired Bootstrap, 100 SPIQA QAs/row): To move beyond aggregate means, we report exact +gained / -lost counts and 95% paired bootstrap CIs:

Platform & BucketComparisonBase / MethodΔ\Delta (Gain/Loss)95% Paired CI
vLLM Llama 4KGS Top-3 vs. Baseline85% / 85%0.0% (+1/-1)[-3.0%, +3.0%]
vLLM Llama 32KGS Top-3 vs. Baseline83% / 87%+4.0% (+5/-1)[0.0%, +9.0%]
vLLM Llama 128KGS Top-3 vs. Baseline81% / 80%-1.0% (+3/-4)[-6.0%, +4.0%]
Native RAPID 32KGS Top-3 vs. RAPID86% / 86%0.0% (+3/-3)[-5.0%, +5.0%]
Native MagicDec 32KGS Top-1 vs. Dense84% / 81%-3.0% (+0/-3)[-7.0%, 0.0%]

(For completeness, non-bootstrapped probes: Native RAPID Top-1 is 84% vs. 86% (+1/-3); DirectKV 32K full/sel is 90%/88% vs. 90%.)

Across vLLM and native RAPID, paired Δ\Delta stays within -1.0% to +4.0% with 95% CIs straddling zero ([-6.0%, +9.0%]), quantifying paired QA-sampling uncertainty over 100 items and confirming no systematic degradation (not a multi-seed non-inferiority test). When strict Top-1 is desired, G1 & G3 also show GraphSpec Top-1 hits 85.8–111.1 TPS on Llama-3.1-8B (3.17x–9.30x vs. EAGLE-3 at 32K–128K; 1.07x–1.85x over official RAPID/SnapKV).

Dear Reviewer EVfm,

Thank you again for your detailed feedback. We have also uploaded the revised PDF (Section 3, RQ5/Table 9, Appendix A.3, and Appendix C.4/Tables 17–18) incorporating the identical-executor selection isolation, unmodified native RAPID (1.46x–1.83x) and MagicDec (1.07x–1.85x) ports, component-level timing breakdowns, Top-1 BF16 non-associativity clarification, and 20,000-replicate paired bootstrap CIs ([-6.0%, +9.0%] straddling zero).

As the author–reviewer discussion period closes on November 18, please let us know if any remaining points need further clarification.

Thank you for the detailed rebuttal and substantial revisions.

The additional experiments, particularly the identical-executor ablation and comparisons with native RAPID and MagicDec implementations, provide stronger evidence for the contribution of knowledge-guided selection. The component-level timing analysis, BF16 numerical clarification, and paired bootstrap results also address my concerns regarding performance attribution and the quality–efficiency trade-off.

Although some limitations remain, especially the modest end-to-end latency improvement and uncertainty regarding quality preservation, I believe my main concerns have been adequately addressed.

I have therefore updated my overall rating from 2 (Weak Reject) to 3 (Weak Accept) and my soundness rating from 2 to 3.

Thank you again for your thorough response.

Reviewer_oPkYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper introduces KAP which is an interface to allow frontend selections to drive the KV access during inference. The paper also provides an instantiation with graphspec where the draft attends to a compact copy of the KV for frontend selected spans from the full prefill and the verification is against a full context verification under Top 1 or Top 3 acceptance.

Top 3 GraphSpec gives 1.17x decode speedup on Qwen and 1.34x on Llama at 128K context length. It is substantially faster than vLLM reimplementations of RAPID and MagicDec SnapKV.

Critical strengths

the observation of the frontend relevance signals being information to help KV access is a good one.

It is not easy to make a sparse-view draft path coexist with vllm's cuda-graph replay and the appendix details are great to read and learn.

this methods transfers across a frontend, a model, a workload, and a verification policy

The paper also shows that task-level priors can select better draft context than relying on attention scores with its comparison to SnapKV.

Critical weaknesses

Top-1 GraphSpec is slower than full-context decoding at every length which means the speedup does not occur there and this is relevant as greedy decoding is often used for reproducibility even if not all gpu ops are deterministic.

I see Top-3 accepts about 95% of drafts so I wonder what happens if you skip verification entirely and decode from the selected KV view directly?

A tps of 50 on 4xA800 for a 32B model probably means single stream decode? If so then the expts are missing batch size and serving realism because the comparison is against snapKV's kv-sparse self-specualtion which performs better with a batch.

The "universal IR" and "compiler" claims are not supported in my opinion. This is because the structured priors that motivate the paper (relations, edges, freshness, reliability) do not survive into execution and the supposed "compiler" performs span alignment and page bookkeeping but no optimization or lowering decisions.

Questions for authors

What is the quality and speed of decoding directly from the selected-KV view without verification? What are B, the number of queries, the batch size, and the decoding temperature? Why does s differ between Top-1 and Top-3?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oPkY: DirectKV (No-Verify), Concurrency, Hyperparameters, and Compiler Lowering

We thank Reviewer oPkY for highlighting our frontend-to-KV insight, CUDA-graph integration, and cross-configuration transfers (see Global Response G1–G5 for full tables).

1. Skipping Verification: Quality & Speed of DirectKV (W2, Q1)

Setting verification policy ρverify=none\rho_{\text{verify}}=\text{none} in KAP lowers the same frontend plan into two position-preserving DirectKV execution paths (G2): full_prefill (retains full-context prefill contextualization, then executes page-aligned in-place KV compaction while preserving original RoPE/M-RoPE pos) and selective_prefill (lowers structured priors directly into a sparse prefill-and-decode schedule that maintains non-contiguous pos continuity across prefill and decode so distractors are never materialized). On Qwen3-VL-32B (TP=4, SPIQA 32K, 100 random QAs, s=15.3%s=15.3\% from identical frozen frontend outputs to control variables):

  • Full-Context Baseline: 100% KV, 52.25 TPS, 4.48s TTFT, 6.79s E2E, 90.0% pass.
  • DirectKV (full_prefill): 15.3% KV, 56.18 TPS, 4.13s TTFT, 6.17s E2E, 90.0% pass (0 net change).
  • DirectKV (selective_prefill): 15.3% KV, 59.52 TPS, 0.51s TTFT (8.8x), 2.48s E2E (2.74x), 88.0% pass (+1/-3).

Verified GraphSpec provides full-context target verification at every cycle—guarding answer quality against any frontend omission (matching full-context quality in Table 4/8 and G5)—while DirectKV offers a high-throughput, low-memory operating point when frontend precision is high.

2. Batch Size, Concurrent Serving, and SnapKV Comparison (W3, Q2)

Yes—primary sweeps used batch_size=1 to isolate per-request latency. Under continuous batching on Qwen3-VL-32B (TP=4, G4) vs. Official vLLM v0.16.0 Baseline:

  • Verified GraphSpec Top-3 beats Baseline across all 32K concurrencies c=2,4,8c=2,4,8 (100.7/177.1/301.4 vs 99.7/168.2/281.7 TPS, 1.01x–1.07x) and 128K c=2,4c=2,4 (76.0/121.2 vs 68.2/104.2 on SPIQA; 69.7/116.1 vs 59.0/101.2 on MuSiQue, 1.12x–1.18x; 128K c=8c=8 omitted due to 80GB full-KV limits).
  • DirectKV releases target decode KV and scales further with batch: cutting KV-pool use 75.3–86.6% (32K c=8c=8) and 78.9–82.9% (128K c=4c=4) vs. full_prefill, hitting 396.3–412.6 TPS at 32K c=8c=8 (1.46x) and 173.7–194.5 TPS at 128K c=4c=4 (1.87x vs Baseline).

3. Top-1 Speedups (W1)

Per Eq. (2), KVcrit≈(1−h)Wmainh−sKV_{\text{crit}} \approx \frac{(1-h)W_{\text{main}}}{h-s}: on Qwen3-VL-32B, large WmainW_{\text{main}} and ~50% Top-1 acceptance keep Top-1 below break-even. On Llama-3.1-8B-Instruct (Top-1 h≈79%–85%h \approx 79\%\text{--}85\%), GraphSpec Top-1 sustains 85.84–111.10 TPS across 4K–128K (3.17x–9.30x over EAGLE-3 at 32K–128K, G1) and beats Official RAPID (1.46x–1.59x) and Official MagicDec-SnapKV (1.07x–1.85x) on their native stacks (G3).

4. Exact BB, Queries, Batch Size, Temperature, and Why ss Differs Between Top-1 & Top-3 (Q2)

  • Queries: Main sweeps use 586 QAs/length on SPIQA and 200 QAs/length on MuSiQue; due to rebuttal time constraints, new sweeps use 100 randomly sampled QAs/bucket.
  • Settings: Main sweeps use batch_size=1, temperature=0, seed=0, max_tokens=1024, k=6k=6.
  • Budget BB: Context-scaled token budget (~1,500–7,500 tokens including protected anchors), yielding ~39% (4K) down to ~5.5%–7.8% (128K) on SPIQA (29.4% to 5.1% on MuSiQue).
  • Why ss differs (7.8% vs. 5.5% at 128K): For any request ii, the compiled plan are 100% identical between Top-1/3, but Table 8 reports output-token-weighted ss (Appendix C.1, Line 838). Different output lengths shift the weighted average; unweighted prompt ratio is identical (7.77% at 128K). We will report both.

5. "Universal IR" & "Compiler" Claims (W4)

Like a compiler IR abstracting source ASTs while preserving execution invariants, KAP avoids coupling kernels to RAG schemas yet resolves both concerns (G5(3)):

  1. Preserved Prior Invariants: A=(U,M,pos,ρaccess,ρverify)\mathcal{A}=(U,M,pos,\rho_{\text{access}},\rho_{\text{verify}}) preserves (i) semantic/provenance boundaries (UU keeps protected anchors and non-contiguous graph spans separate without merging unrelated regions), (ii) non-contiguous original RoPE/M-RoPE pos, and (iii) prior-confidence policies (ρaccess,ρverify\rho_{\text{access}},\rho_{\text{verify}} deciding verified dual-view vs. direct DirectKV/selective_prefill).
  2. Compiler Lowering & Optimizations: Across 3 heterogeneous KV memory models (vLLM PagedAttention, HF AppendCache, FlashInfer Paged Cache), our compiler executes 3 lowering/optimization decisions: arena-bucket & q=1/q=k+1 CUDA-graph selection, page-growth & coalesced-sync scheduling, and policy-driven lifecycle lowering. We pair "universal IR" with Cross-Layer Runtime Access Contract.

Dear Reviewer oPkY,

Thank you again for your constructive feedback. We have also uploaded the revised PDF (Section 2, Section 3, RQ5/Table 9, Appendix A.5, and Appendix C.4/Table 17) incorporating all new results—including verification-free DirectKV (90.0% pass in full-prefill; 8.8x TTFT / 2.74x E2E in selective-prefill), continuous-batching concurrency vs. official vLLM (1.46x–1.87x TPS, 75.3%–89.0% KV-pool savings, 34.6–149.8s -> 2.5–5.7s p50 latency), Llama-3.1-8B Top-1 speedups (3.17x–9.30x vs. EAGLE-3), and the 3-backend lowering details.

As the author–reviewer discussion period closes on November 18, please let us know if any remaining questions need further clarification.

Thank you for the rebuttal. The hyperparameters, batch size, retained KV-accounting, paired CIs and native MagicDec make the contributions more credible. I am raising my score to 3 and not a 4 because:

  • Verified GraphSpec only speeds things up at long context (the results ar 4k context at various batch sizes show this)
  • GraphSpec's verification shows not a lot of benefit as evidenced by the results above and DirectKV is close enough at higher speed
  • I do not agree with the "universal IR" and "compiler" framing as the structured priors still do not influence execution, and the lowering decisions are runtime engineering choices rather than something in the framework

In general, the paper's useful finding is that frontend-selected KV works well as decoding context

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: EAGLE-3 & Top-1 Speedups, DirectKV (No-Verify), Native RAPID/MagicDec Ports, Concurrency, and Paired CIs / Compiler Clarification

We thank all reviewers for their feedback. Due to rebuttal time constraints, new experiments use 100 random QAs/bucket unless noted.

G1. EAGLE-3 & Top-1

No EAGLE-3 exists for Qwen3-VL-32B; on Llama-3.1-8B (vLLM v0.16, TP=2, greedy, max=512) vs. RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3:

BucketEAGLE-3GS Top-1(vs E3)GS Top-3(vs E3)
SPIQA 4K84.7(29%)107.6(85%/1.27x)124.3(97%/1.47x)
SPIQA 32K27.2(3.7%)104.0(83%/3.83x)116.7(96%/4.29x)
SPIQA 128K9.2(1.8%)85.8(79%/9.30x)99.1(95%/10.73x)
MuSiQue 4K154.5(31%)111.1(85%/0.72x)122.6(97%/0.79x)
MuSiQue 32K32.6(3.5%)103.4(83%/3.17x)113.7(96%/3.49x)
MuSiQue 128K9.9(2.8%)87.0(81%/8.77x)92.4(95%/9.31x)

EAGLE-3 drops from ~30% acceptance (4K) to 1.8–3.7% (32K–128K); GraphSpec sustains 79–85% (Top-1) and 95–97% (Top-3), beating EAGLE-3 by 3.17x–9.30x (Top-1) and 3.49x–10.73x (Top-3).

G2. DirectKV (oPkY, dENe)

Setting verification policy ρverify=none\rho_{\text{verify}}=\text{none}, full_prefill frees unselected KV post-prefill and selective_prefill keeps prefill+decode sparse. On Qwen3-VL-32B (TP=4, SPIQA 32K, s=15.3%s=15.3\% via identical frozen frontend outputs to control variables):

MethodKV%TPSTTFTE2EPass
Full-Context100%52.254.48s6.79s90%
DirectKV(full)15.3%56.184.13s6.17s90%
DirectKV(sel)15.3%59.520.51s2.48s(2.74x)88%

G3. Native Ports (32K) (dENe, EVfm, UKSv)

We kept official RAPID (HF+FA2) & MagicDec (FlashInfer) intact (SHA-256/AST verified) and ported KAP on Llama-3.1-8B (1x A800, 32K; RAPID tests 8B on 1x A800, MagicDec supports it in-repo; identical frozen selections 15.0%/8.8% to control variables). SnapKV uses strict Top-1 while RAPID uses probabilistic acceptance, so we test Top-1 on MagicDec and Top-1/3 on RAPID:

Stack/DataOfficial TPS (QA time, KV%)GS Top-1 / Top-3 TPS (QA time, speedup)DirectKV full/sel TPS (QA time, sel speedup)
RAPID SPIQA19.4 (9.48s, 10.5%)30.8 (7.02s, 1.59x) / 35.6 (6.67s, 1.83x)66.6/66.9 (6.2s/2.3s, 3.45x)
RAPID MuSiQue20.9 (6.63s, 10.5%)30.4 (5.57s, 1.46x) / 35.7 (5.50s, 1.71x)68.2/68.5 (5.2s/1.5s, 3.28x)
MagicDec SPIQA36.2 (9.04s, 13.2%)66.8 (7.79s, 1.85x)80.5/81.4 (7.2s/2.8s, 2.25x)
MagicDec MuSiQue61.4 (7.58s, 13.1%)65.7 (7.41s, 1.07x)81.7/81.7 (6.7s/2.5s, 1.33x)

G4. Concurrency (oPkY, UKSv)

Qwen3-VL-32B (TP=4) at c∈{2,4,8}c\in\{2,4,8\} (4K/32K) & c∈{2,4}c\in\{2,4\} (128K; c=8c=8 OOMs full KV on 80GB) vs. Official vLLM v0.16.0 Baseline (TPS):

Bucket(cc)BaselineGS Top-3DirectKV(full)DirectKV(sel)
SPIQA 4K(2/4/8)116.4/220.2/391.2111.0/208.2/375.1117.5/222.7/409.2117.4/225.1/410.4(1.05x)
MuSiQue 4K(2/4/8)118.7/227.3/413.0112.1/207.3/378.8127.0/226.8/430.6128.1/242.7/436.9(1.06x)
SPIQA 32K(2/4/8)99.7/168.2/281.7100.7/177.1/301.4(1.07x)109.7/208.9/396.3114.9/222.6/412.6(1.46x)
MuSiQue 32K(2/4/8)102.4/170.8/270.1108.3/174.3/276.6(1.02x)115.0/203.2/389.4117.2/216.5/410.6(1.52x)
SPIQA 128K(2/4)68.2/104.276.0/121.2(1.16x)99.7/173.7(1.67x)105.7/194.5(1.87x)
MuSiQue 128K(2/4)59.0/101.269.7/116.1(1.15x)79.1/131.0(1.29x)103.6/161.6(1.60x)

At 32K–128K, GS3 beats Baseline at all cc (1.01x–1.18x, 16.0%/5.5–7.8% KV). DirectKV(sel) cuts KV-pool use 75.3–86.6% (32K c=8c=8) & 78.9–82.9% (128K c=4c=4) vs. full, hits 1.15x–1.87x TPS, and cuts p50 latency 34.3–34.6s->3.2–5.6s (32K) & 147.5–148.1s->3.8–5.7s (128K).

G5. CIs, Top-1 Scope & Lowering

  1. Paired CIs (20k bootstrap, 100 SPIQA QAs): vLLM Llama GS3 vs. Baseline: 4K 0%([-3,+3]), 32K +4%([0,+9]), 128K -1%([-6,+4]); RAPID GS3: 0%([-5,+5]); MagicDec GS1 vs. SnapKV: 81 vs 84% (1.85x TPS).
  2. Top-1 Scope: Top-1 enforces the exact greedy rule per step; BF16 reductions across q=1q=1 vs. q=k+1q=k+1 can flip near-tie logits. Answer evals confirm Top-1 preserves quality (84 vs 86% RAPID, 81 vs 84% MagicDec, ~1–2% on Qwen 128K).
  3. Prior Invariants & Compiler Lowering: Like a compiler IR abstracting source ASTs while preserving execution invariants, KAP avoids coupling kernels to RAG schemas yet preserves 3 invariants in A=(U,M,pos,ρaccess,ρverify)\mathcal{A}=(U,M,pos,\rho_{\text{access}},\rho_{\text{verify}}): (i) semantic/provenance boundaries (UU keeps protected anchors and non-contiguous graph spans separate without merging unrelated regions), (ii) non-contiguous original RoPE/M-RoPE pos, and (iii) prior-confidence policies (ρaccess,ρverify\rho_{\text{access}},\rho_{\text{verify}} deciding verified dual-view vs. direct DirectKV/selective_prefill). Across 3 heterogeneous KV memory models (vLLM PagedAttention, HF AppendCache, FlashInfer Paged Cache), our compiler executes 3 lowering/optimization decisions: arena-bucket & CUDA-graph selection, page-growth & coalesced-sync scheduling, and policy-driven lifecycle lowering. We pair "universal IR" with Cross-Layer Runtime Access Contract.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.