- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work targets the storage overhead incurred by duplicated document KV states in RAG workloads featuring heavy cross-request reuse. Each document chunk is encoded into a shared, immutable INT4 base representation, while request-local INT4 replacement overlays are introduced to recover partial contextual interactions. During inference, a single global softmax operates jointly over the unmodified base entries, replacement overlays, and dynamic BF16 KV states. This design eliminates the need to persist a full dense KV cache for every individual request. The paper’s strongest and most robust contribution lies in unifying storage and execution for low-bit shared states paired with private repair states. By contrast, query-guided recomputation, single-copy sharing, replacement masking, and blockwise softmax are techniques with well-established prior art.
Critical strengths
The paper’s key strength is its well-specified cache design combining shared low-bit document states, request-private replacement overlays, and globally normalized attention. Rigorous ablations separate storage-sharing gains from quantization and repair-selection effects, and storage audits confirm requests reuse canonical base objects instead of storing duplicates. Its composite attention operator includes a scoped equivalence proof, numerically validated with a maximum absolute discrepancy of and consistent outputs across all 180 generation test points. Evaluations over four models and 13 LongBench tasks show major persistent KV savings, with macro-average quality closely matching the full-precision baseline.
Critical weaknesses
-
FusionRAG already incorporates query-guided recomputation, shared document KV, private replacement pages, and masking for obsolete entries. Accordingly, OneBaseKV’s core contributions narrow down to low-bit representation and the associated execution integration. The manuscript should better articulate the technical hurdles addressed by this integration and furnish stronger evidence showing its advantages surpass a trivial low-bit adaptation of the existing FusionRAG framework. The architecture-normalized selector adds little distinguishing value, as its head aggregation reduces to standard query-head averaging across all tested models.
-
Insufficient evaluation against closely related baselines. ProphetKV and InfoFlow KV, two ICML 2026 publications, are mentioned yet omitted from the primary evaluation suite. The storage-matched selector benchmark focuses mainly on CacheBlend, leaving the most competitive query-aware baselines unexamined. In addition, the serving-side INT4 ablations maintain full per-request caches instead of replicating OneBaseKV’s shared replacement structure. This setup cannot cleanly separate gains unique to the proposed method from simple combinations of state sharing and quantization.
-
Observed serving advantages hold only within constrained resource and workload settings. The capacity analysis caps persistent KV consumption but excludes roughly 6.6 GiB of transient memory overhead. Although goodput gains are reported, merely 10.4%–13.9% of finished requests satisfy the 600 ms TTFT requirement, and overall completion rates stay poor. These results demonstrate relative improvements under the chosen congested workloads, yet fail to prove practical capacity gains under a fixed total GPU memory budget paired with acceptable service-level objectives. Additional load sweeps, evaluation across multiple SLO thresholds, and longer-generation experiments are required.
Questions for authors
How does OneBaseKV perform against FusionRAG, ProphetKV, and InfoFlow KV when all methods adopt the same INT4 shared-base and private-overlay representation, identical repair budget, and unified attention execution engine? Does the proposed selector still yield measurable gains in quality or latency under these controlled conditions?
4 comments
W2 — Matched ProphetKV and InfoFlow KV comparisons in initialization and serving
Thank you for recommending comparisons with closely related query-aware methods under matched storage and execution conditions.
We test query-aware baselines and the combination of sharing with quantization. FusionRAG, ProphetKV, InfoFlow KV, CacheBlend and OneBaseKV selection rules use the same shared INT4 base, request-private INT4 replacement overlays, 30% repair budget and attention executor. This is a controlled low-bit adaptation of the external selection rules. The complete per-request INT4 controls are separate storage configurations. The matched controls directly test what the selection rule contributes after sharing, precision and execution have been fixed.
For each model, all rules use the same 100 HotpotQA and 100 Qasper examples. Persistent bytes match exactly per example, including quantized payloads, quantization metadata and overlay indices. The table summarizes the requested methods and the FusionRAG combination control. Relative persistent KV is normalized to OneBaseKV per example and does not denote whole-device peak memory.
| Selection rule on the common INT4 cache | Macro quality | 95% confidence interval | Relative persistent KV |
|---|---|---|---|
| OneBaseKV | 50.94 | [46.17, 55.75] | 1.000 |
| FusionRAG | 47.53 | [42.61, 52.41] | 1.000 |
| ProphetKV | 51.86 | [46.97, 56.73] | 1.000 |
| InfoFlow KV | 50.40 | [45.55, 55.22] | 1.000 |
Under these matched conditions, OneBaseKV gains 3.41 macro-quality points over FusionRAG selection with a paired 95% interval of [1.41, 5.45]. The paired intervals against ProphetKV and InfoFlow KV are [−2.42, 0.61] and [−0.84, 1.94], both including zero. Our W1 and Q1 responses give the complete combination and selection comparisons.
The following stage medians are in ms. Probe measures selection only. Full initialization includes probing, repair and packing, so the two times are not added.
| Model and task | Stage | OneBaseKV | ProphetKV | InfoFlow KV |
|---|---|---|---|---|
| Qwen3-8B, HotpotQA | Probe | 54.4 | 82.0 | 60.9 |
| Qwen3-8B, HotpotQA | Full initialization | 948.5 | 978.7 | 956.3 |
| Qwen3-8B, Qasper | Probe | 38.1 | 59.4 | 42.3 |
| Qwen3-8B, Qasper | Full initialization | 330.9 | 363.5 | 345.4 |
| Llama-3.1-8B, HotpotQA | Probe | 46.8 | 67.2 | 52.1 |
| Llama-3.1-8B, HotpotQA | Full initialization | 855.7 | 877.6 | 862.0 |
| Llama-3.1-8B, Qasper | Probe | 33.4 | 51.5 | 38.7 |
| Llama-3.1-8B, Qasper | Full initialization | 296.7 | 311.2 | 309.4 |
Among these three methods, OneBaseKV has the lowest median at both stages in all four settings. Its probe time is approximately 30.4% to 35.9% lower than ProphetKV's and 9.9% to 13.7% lower than InfoFlow KV's. Full initialization time is 2.5% to 9.0% and 0.7% to 4.2% lower, respectively. OneBaseKV probes two relative depths, ProphetKV retains all-layer scoring and InfoFlow KV uses four layers with fixed chunk order. Full initialization also includes the common repair and packing work. Across five held-out queries and five selection rules, all 25 selected-index sets match those actually supplied to overlay packing.
In the separate service experiment, ProphetKV and InfoFlow KV also use shared INT4 bases and private INT4 overlays built from their own online selections, with the same 30% repair budget and executor. At 32 output tokens and 0.1114 arrivals/s, OneBaseKV and InfoFlow KV meet the 24 GiB budget in all seeds, with 600 ms goodput rates of 0.011463 and 0.011073 req/s, respectively, while ProphetKV exceeds the budget in at least one run at this load. Across five loads, OneBaseKV fits the budget in all 15 runs, ProphetKV in 11 and InfoFlow KV in 12. Our W3 response reports service results across loads and deadlines.
In the submitted revision, we have added the matched selection comparison with ProphetKV and InfoFlow KV to the final paragraph of Section 4.4. Tables A9 and A10 in Appendix B.1 report quality and initialization timings at identical storage bytes and repair budgets. The second paragraph of Appendix C.5 specifies that both selection rules actually drive shared INT4 bases and private replacement overlays. The paragraph immediately after Table A19 reports their service and budget results.
W3 — Load, output length and deadline sensitivity under a total device budget
Thank you for suggesting tests across arrival rates, deadlines and output lengths.
The original 10.4% to 13.9% for OneBaseKV and 2.6% to 2.8% for Fusion-style are fractions of completed requests meeting 600 ms, not relative speedups. The original workloads test service under capacity pressure. We vary arrival load, TTFT deadline and output length under a 24 GiB whole-device budget, with controlled Fusion-style as the primary reference and matched query-aware methods as additional comparisons.
On one RTX 5090, NF4 Qwen3-8B uses 12K contexts. Per seed, each method replays the same 192 timestamped arrivals over 32 documents and 64 sessions. NVML peaks include weights, KV and temporary workspace. TTFT includes queueing from arrival. Outputs increase from 32 to 128 tokens. Arrival rate measures offered requests per second.
Below are three-seed mean window goodputs in req/s. Requests must finish in-window and meet the TTFT limit. † marks at least one run above 24 GiB. Bold marks the higher rate in each pair.
| Output tokens | Arrivals/s | Fusion-style, 600 ms | OneBaseKV, 600 ms | Fusion-style, 3000 ms | OneBaseKV, 3000 ms |
|---|---|---|---|---|---|
| 32 | 0.0371 | 0.001046 | 0.005045 | 0.008125 | 0.032689 |
| 32 | 0.1114 | 0.000971 | 0.011463 | 0.010285 | 0.070361 |
| 32 | 0.1857 | 0.000324 | 0.006829 | 0.001311 | 0.058269 |
| 128 | 0.0371 | 0.000719† | 0.003932 | 0.005767† | 0.023797 |
| 128 | 0.1114 | 0.000193 | 0.000584 | 0.001165 | 0.002722 |
At 32 tokens and 600 ms, OneBaseKV retains higher goodput than Fusion-style as arrivals increase across all three rates. With four times longer output, it remains higher at both rates and deadlines. At 128 output tokens and 0.1114 arrivals/s, the overall window completion rates are 0.08999 requests/s for OneBaseKV and 0.08493 for Fusion-style, counting all in-window completions regardless of TTFT.
OneBaseKV fits 24 GiB in all 15 runs and Fusion-style in 14. ProphetKV and InfoFlow KV exceed it in four and three runs, respectively. For each load, we compare only methods that remain within 24 GiB in all three seeds at that load. Under this load-specific comparison, OneBaseKV has the highest goodput at four of five loads under both deadlines, including both 128-token loads.
The eight-deadline open-arrival sweep below averages the within-seed OneBaseKV/Fusion-style goodput ratios. Both fit 24 GiB in every seed at these two 32-token loads. These statistics are separate from the original serial-trace deadline sweep.
| TTFT ms | 0.0371 arrivals/s | 0.1114 arrivals/s |
|---|---|---|
| 300 | 4.87× | 12.50× |
| 450 | 4.87× | 12.67× |
| 600 | 4.87× | 12.67× |
| 900 | 4.48× | 13.17× |
| 1200 | 16.34× | 41.00× |
| 1800 | 2.61× | 4.98× |
| 3000 | 4.56× | 8.37× |
| 6000 | 1.28× | 2.82× |
OneBaseKV has higher goodput at all eight deadlines and both arrival rates.
For absolute attainment at 32 tokens and 0.1114 arrivals/s, we follow all 576 arrivals through a further 120 s drain. Counts meeting 600 ms are OneBaseKV 60, InfoFlow KV 58 and Fusion-style five. ProphetKV exceeds 24 GiB in at least one run at this load. These counts include drain completions and are separate from window goodput.
At 32 tokens and 0.0371 arrivals/s, all three methods below fit 24 GiB in every seed. Entries give qualifying completions through the window and 120 s drain, and percentages. The denominator is all 576 arrivals.
| TTFT ms | Fusion-style | ProphetKV | OneBaseKV |
|---|---|---|---|
| 600 | 18, 3.13% | 78, 13.54% | 79, 13.72% |
| 3000 | 126, 21.88% | 497, 86.28% | 503, 87.33% |
| 6000 | 443, 76.91% | 554, 96.18% | 555, 96.35% |
At 32 tokens and 0.0371 arrivals/s, OneBaseKV serves 96.35% of all arrivals within the 6000 ms TTFT deadline and the stated completion cutoff, identifying a service operating point at the longer deadline.
The submitted revision adds whole-device-budget service results to Section 4.3's final paragraph. In Appendix C.5, Table A19 compares arrival rates, TTFT deadlines and 32- and 128-token outputs. Table A20(a) gives the eight-deadline sweep at two arrival rates and A20(b) gives absolute attainment. C.5's third paragraph and the attainment paragraphs before Table A20 distinguish overall window completion rates, window goodput and attainment among all arrivals.
Q1 — Quality and latency when only the selector changes
We appreciate your suggestion to isolate selector effects using the same cache representation, repair budget and execution engine.
We compared five selection rules under the requested controls. Each model uses the same 100 HotpotQA questions and 100 Qasper questions for every rule. All methods use the same shared INT4 base, private INT4 replacement format, 30% repair budget and attention executor. Persistent bytes match per example. The model weights in this quality experiment are BF16. The KV representation being compared is INT4 for every selector.
The table reports macro quality across the four model and task combinations. The intervals are for each selector's score.
| Selector | Macro quality (points) | 95% confidence interval |
|---|---|---|
| OneBaseKV | 50.94 | [46.17, 55.75] |
| FusionRAG | 47.53 | [42.61, 52.41] |
| ProphetKV | 51.86 | [46.97, 56.73] |
| InfoFlow KV | 50.40 | [45.55, 55.22] |
| CacheBlend | 49.82 | [44.99, 54.64] |
For the paired comparison, we resample documents within tasks 10,000 times and keep methods paired on the same examples. OneBaseKV gains 3.41 points over FusionRAG selection, with a 95% interval of [1.41, 5.45]. Against ProphetKV the difference is −0.92 points with a 95% interval of [−2.42, 0.61]. Against InfoFlow KV it is +0.54 points with a 95% interval of [−0.84, 1.94]. Both of the latter paired intervals include zero, so these data do not resolve a directional quality difference with either selector.
The following table separates median probe time from full initialization time, both in milliseconds. Full initialization includes the probe, repair and packing.
| Model and task | Stage | OneBaseKV | FusionRAG | ProphetKV | InfoFlow KV | CacheBlend |
|---|---|---|---|---|---|---|
| Qwen3-8B, HotpotQA | Probe | 54.4 | 67.2 | 82.0 | 60.9 | 2216.7 |
| Qwen3-8B, HotpotQA | Full initialization | 948.5 | 960.0 | 978.7 | 956.3 | 3116.7 |
| Qwen3-8B, Qasper | Probe | 38.1 | 42.1 | 59.4 | 42.3 | 625.6 |
| Qwen3-8B, Qasper | Full initialization | 330.9 | 345.6 | 363.5 | 345.4 | 925.8 |
| Llama-3.1-8B, HotpotQA | Probe | 46.8 | 60.2 | 67.2 | 52.1 | 2541.4 |
| Llama-3.1-8B, HotpotQA | Full initialization | 855.7 | 867.4 | 877.6 | 862.0 | 3355.6 |
| Llama-3.1-8B, Qasper | Probe | 33.4 | 36.7 | 51.5 | 38.7 | 720.8 |
| Llama-3.1-8B, Qasper | Full initialization | 296.7 | 289.2 | 311.2 | 309.4 | 980.8 |
Among all five rules, OneBaseKV has the lowest median probe time in all four combinations and the lowest median full initialization time in three. Across these four settings its median full initialization time is 64.3% to 74.5% lower than CacheBlend's, and its 95th-percentile full initialization time is 2.3% to 5.8% lower than ProphetKV's. The common representation and executor preserve distinct selector effects. OneBaseKV has higher quality than FusionRAG selection and lower probe and full initialization medians than ProphetKV and InfoFlow KV throughout the comparison.
In the submitted revision, we have added the quality comparison of five selection rules under a common cache representation, repair budget and executor to the final paragraph of Section 4.4. Table A9 in Appendix B.1 reports paired quality-difference confidence intervals, and Table A10 provides probe and full initialization timings for each model and task. The second paragraph of Appendix C.3 clarifies the distinction between model-weight and KV-storage precision.
W1 — What OneBaseKV adds to FusionRAG
We appreciate the opportunity to explain OneBaseKV's technical differences from FusionRAG and examine them through controlled comparisons.
Both methods share document KV, select repairs and mask replaced entries. OneBaseKV differs in how it builds, stores and executes these states. The integration must preserve logical positions and replacement masking across KV streams with different precisions under a common softmax denominator, without reintroducing a persistent dense request cache. FusionRAG builds chunk KV with retrieved neighbors and selects repairs from final-layer scores. OneBaseKV prefills each chunk independently, keeps its immutable INT4 base once across requests, and probes at two relative layer depths. The overlay, mask and dynamic history are request-private. At decode time, the unreplaced INT4 base entries, private INT4 replacements and BF16 history retain their logical positions and share one softmax denominator. This avoids a persistent dense request cache. Proposition 1 in the paper establishes equivalence to materializing identical dequantized KV. Proposition 2 bounds quantization perturbation with different base and overlay precisions. Head grouping equals the query-head mean on our uniform GQA models. We do not attribute the measured gains to head grouping itself.
Only selection varies in the first comparison. Both rules use the same shared INT4 base, private INT4 overlay format, 30% repair budget and attention executor. The table gives per-model and per-task scores and macro quality.
OneBaseKV scores higher in all four settings, with a paired macro gain of 3.41 points and a 95% confidence interval of [1.41, 5.45].
Fusion-style denotes the shared-BF16 control. Table A1 in the paper compares whole configurations in a separate experiment. Against Fusion-style selection with a full per-request INT4 cache, OneBaseKV has higher macro quality with 40.8% less persistent KV at 12K and fanout 16. KV/COW is relative to the two-model mean persistent KV for Fusion-style. Bold identifies low-bit column optima.
The additional ownership control fixes INT4 precision and repair positions. With two documents and four sessions fully resident, storing each base once rather than replicating it per session saves 1,296 MiB of peak persistent KV. This isolates ownership from selection and bit width.
In the submitted revision, we have added the selection comparison under common low-bit storage and execution to the final paragraph of Section 4.4. Table A9 in Appendix B.1 provides detailed selection results. Table A11 and its preceding paragraph in B.1 report the ownership ablation at fixed precision and repair positions.