acceptodds
Under review as a conference paper at ICLR 2027

OneBaseKV: Low-Bit Canonical KV Sharing with Per-Request Replacement Overlays for High-Fanout RAG

23 views

Abstract

Long-context services increasingly process recurring content across many requests. High-fanout retrieval-augmented generation (RAG) is a representative workload in which the same document chunks are repeatedly combined with different queries, orders, and retrieval results. Autoregressive inference retains key–value (KV) states for these long inputs, so per-request cache ownership replicates document state and rapidly consumes memory even after low-bit quantization. Exact prefix sharing removes duplication only for byte-identical prefixes, while cache-fusion systems do not encode shared document state and query-conditioned repairs as jointly executable low-bit objects. We propose OneBaseKV, which stores each reusable chunk once as an immutable INT4 canonical base and represents query-sensitive differences with private INT4 replacement overlays. An architecture-normalized selector identifies repair positions, and materialization-equivalent composite attention merges the base, overlay, and dynamic KV through one global online softmax without a persistent dense request cache. We prove exact equivalence to standard attention over the corresponding materialized dequantized KV and derive a mixed-partition perturbation bound for heterogeneous precisions. Experiments show that OneBaseKV achieves the best quality among the compared efficient methods under fixed persistent-KV budgets. At a BF16 copy-on-write (BF16-COW) persistent-KV cap, it exceeds an oracle choosing the best feasible external method for each combination of model and task by 5.48%. At a 12K-token context and fanout 16, OneBaseKV uses only 37.7% of BF16-COW persistent KV. In two fixed-capacity serving traces, it achieves – the service-level objective (SLO) goodput of Fusion-style. On non-retrieval RULER at 32K, it exceeds every efficient external method on three of four models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
1. Retrieval-Augmented Generation(RAG) 2. KV Cache Sharing 3. KV Cache Quantization 4. Query-Aware Cache Reuse 5. Large Language Model Serving

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_pYZROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets the storage overhead incurred by duplicated document KV states in RAG workloads featuring heavy cross-request reuse. Each document chunk is encoded into a shared, immutable INT4 base representation, while request-local INT4 replacement overlays are introduced to recover partial contextual interactions. During inference, a single global softmax operates jointly over the unmodified base entries, replacement overlays, and dynamic BF16 KV states. This design eliminates the need to persist a full dense KV cache for every individual request. The paper’s strongest and most robust contribution lies in unifying storage and execution for low-bit shared states paired with private repair states. By contrast, query-guided recomputation, single-copy sharing, replacement masking, and blockwise softmax are techniques with well-established prior art.

Critical strengths

The paper’s key strength is its well-specified cache design combining shared low-bit document states, request-private replacement overlays, and globally normalized attention. Rigorous ablations separate storage-sharing gains from quantization and repair-selection effects, and storage audits confirm requests reuse canonical base objects instead of storing duplicates. Its composite attention operator includes a scoped equivalence proof, numerically validated with a maximum absolute discrepancy of �0��0� and consistent outputs across all 180 generation test points. Evaluations over four models and 13 LongBench tasks show major persistent KV savings, with macro-average quality closely matching the full-precision baseline.

Critical weaknesses
  1. FusionRAG already incorporates query-guided recomputation, shared document KV, private replacement pages, and masking for obsolete entries. Accordingly, OneBaseKV’s core contributions narrow down to low-bit representation and the associated execution integration. The manuscript should better articulate the technical hurdles addressed by this integration and furnish stronger evidence showing its advantages surpass a trivial low-bit adaptation of the existing FusionRAG framework. The architecture-normalized selector adds little distinguishing value, as its head aggregation reduces to standard query-head averaging across all tested models.

  2. Insufficient evaluation against closely related baselines. ProphetKV and InfoFlow KV, two ICML 2026 publications, are mentioned yet omitted from the primary evaluation suite. The storage-matched selector benchmark focuses mainly on CacheBlend, leaving the most competitive query-aware baselines unexamined. In addition, the serving-side INT4 ablations maintain full per-request caches instead of replicating OneBaseKV’s shared replacement structure. This setup cannot cleanly separate gains unique to the proposed method from simple combinations of state sharing and quantization.

  3. Observed serving advantages hold only within constrained resource and workload settings. The capacity analysis caps persistent KV consumption but excludes roughly 6.6 GiB of transient memory overhead. Although goodput gains are reported, merely 10.4%–13.9% of finished requests satisfy the 600 ms TTFT requirement, and overall completion rates stay poor. These results demonstrate relative improvements under the chosen congested workloads, yet fail to prove practical capacity gains under a fixed total GPU memory budget paired with acceptable service-level objectives. Additional load sweeps, evaluation across multiple SLO thresholds, and longer-generation experiments are required.

Questions for authors

How does OneBaseKV perform against FusionRAG, ProphetKV, and InfoFlow KV when all methods adopt the same INT4 shared-base and private-overlay representation, identical repair budget, and unified attention execution engine? Does the proposed selector still yield measurable gains in quality or latency under these controlled conditions?

4 comments
AuthorsOpenReview ↗

W1 — What OneBaseKV adds to FusionRAG

We appreciate the opportunity to explain OneBaseKV's technical differences from FusionRAG and examine them through controlled comparisons.

Both methods share document KV, select repairs and mask replaced entries. OneBaseKV differs in how it builds, stores and executes these states. The integration must preserve logical positions and replacement masking across KV streams with different precisions under a common softmax denominator, without reintroducing a persistent dense request cache. FusionRAG builds chunk KV with retrieved neighbors and selects repairs from final-layer scores. OneBaseKV prefills each chunk independently, keeps its immutable INT4 base once across requests, and probes at two relative layer depths. The overlay, mask and dynamic history are request-private. At decode time, the unreplaced INT4 base entries, private INT4 replacements and BF16 history retain their logical positions and share one softmax denominator. This avoids a persistent dense request cache. Proposition 1 in the paper establishes equivalence to materializing identical dequantized KV. Proposition 2 bounds quantization perturbation with different base and overlay precisions. Head grouping equals the query-head mean on our uniform GQA models. We do not attribute the measured gains to head grouping itself.

Only selection varies in the first comparison. Both rules use the same shared INT4 base, private INT4 overlay format, 30% repair budget and attention executor. The table gives per-model and per-task scores and macro quality.

Model and taskFusionRAG selectorOneBaseKV selector
Qwen3-8B, HotpotQA57.1063.33
Qwen3-8B, Qasper37.8940.82
Llama-3.1-8B, HotpotQA52.2456.60
Llama-3.1-8B, Qasper42.9143.02
Macro quality47.5350.94

OneBaseKV scores higher in all four settings, with a paired macro gain of 3.41 points and a 95% confidence interval of [1.41, 5.45].

Fusion-style denotes the shared-BF16 control. Table A1 in the paper compares whole configurations in a separate experiment. Against Fusion-style selection with a full per-request INT4 cache, OneBaseKV has higher macro quality with 40.8% less persistent KV at 12K and fanout 16. KV/COW is relative to the two-model mean persistent KV for Fusion-style. Bold identifies low-bit column optima.

ConfigurationQwen3-8B HotpotQAQwen3-8B QasperLlama-3.1-8B HotpotQALlama-3.1-8B QasperMacroKV (GiB)KV/COW
Fusion-style56.1137.5153.4945.7248.2114.871.000
CacheBlend-style BF1659.3339.9955.8944.5049.9314.891.001
Fusion-style + per-request INT455.9837.6051.8644.5547.509.480.638
CacheBlend-style + per-request INT459.0940.5255.7045.6950.259.480.638
CacheBlend selector + matched B4P459.2340.2255.7044.6649.965.610.377
OneBaseKV63.3340.8256.6042.9050.915.610.377

The additional ownership control fixes INT4 precision and repair positions. With two documents and four sessions fully resident, storing each base once rather than replicating it per session saves 1,296 MiB of peak persistent KV. This isolates ownership from selection and bit width.

Base ownershipPeak persistent KV (MiB)
Replicated per session3,371
Shared per document2,075

In the submitted revision, we have added the selection comparison under common low-bit storage and execution to the final paragraph of Section 4.4. Table A9 in Appendix B.1 provides detailed selection results. Table A11 and its preceding paragraph in B.1 report the ownership ablation at fixed precision and repair positions.

AuthorsOpenReview ↗

W2 — Matched ProphetKV and InfoFlow KV comparisons in initialization and serving

Thank you for recommending comparisons with closely related query-aware methods under matched storage and execution conditions.

We test query-aware baselines and the combination of sharing with quantization. FusionRAG, ProphetKV, InfoFlow KV, CacheBlend and OneBaseKV selection rules use the same shared INT4 base, request-private INT4 replacement overlays, 30% repair budget and attention executor. This is a controlled low-bit adaptation of the external selection rules. The complete per-request INT4 controls are separate storage configurations. The matched controls directly test what the selection rule contributes after sharing, precision and execution have been fixed.

For each model, all rules use the same 100 HotpotQA and 100 Qasper examples. Persistent bytes match exactly per example, including quantized payloads, quantization metadata and overlay indices. The table summarizes the requested methods and the FusionRAG combination control. Relative persistent KV is normalized to OneBaseKV per example and does not denote whole-device peak memory.

Selection rule on the common INT4 cacheMacro quality95% confidence intervalRelative persistent KV
OneBaseKV50.94[46.17, 55.75]1.000
FusionRAG47.53[42.61, 52.41]1.000
ProphetKV51.86[46.97, 56.73]1.000
InfoFlow KV50.40[45.55, 55.22]1.000

Under these matched conditions, OneBaseKV gains 3.41 macro-quality points over FusionRAG selection with a paired 95% interval of [1.41, 5.45]. The paired intervals against ProphetKV and InfoFlow KV are [−2.42, 0.61] and [−0.84, 1.94], both including zero. Our W1 and Q1 responses give the complete combination and selection comparisons.

The following stage medians are in ms. Probe measures selection only. Full initialization includes probing, repair and packing, so the two times are not added.

Model and taskStageOneBaseKVProphetKVInfoFlow KV
Qwen3-8B, HotpotQAProbe54.482.060.9
Qwen3-8B, HotpotQAFull initialization948.5978.7956.3
Qwen3-8B, QasperProbe38.159.442.3
Qwen3-8B, QasperFull initialization330.9363.5345.4
Llama-3.1-8B, HotpotQAProbe46.867.252.1
Llama-3.1-8B, HotpotQAFull initialization855.7877.6862.0
Llama-3.1-8B, QasperProbe33.451.538.7
Llama-3.1-8B, QasperFull initialization296.7311.2309.4

Among these three methods, OneBaseKV has the lowest median at both stages in all four settings. Its probe time is approximately 30.4% to 35.9% lower than ProphetKV's and 9.9% to 13.7% lower than InfoFlow KV's. Full initialization time is 2.5% to 9.0% and 0.7% to 4.2% lower, respectively. OneBaseKV probes two relative depths, ProphetKV retains all-layer scoring and InfoFlow KV uses four layers with fixed chunk order. Full initialization also includes the common repair and packing work. Across five held-out queries and five selection rules, all 25 selected-index sets match those actually supplied to overlay packing.

In the separate service experiment, ProphetKV and InfoFlow KV also use shared INT4 bases and private INT4 overlays built from their own online selections, with the same 30% repair budget and executor. At 32 output tokens and 0.1114 arrivals/s, OneBaseKV and InfoFlow KV meet the 24 GiB budget in all seeds, with 600 ms goodput rates of 0.011463 and 0.011073 req/s, respectively, while ProphetKV exceeds the budget in at least one run at this load. Across five loads, OneBaseKV fits the budget in all 15 runs, ProphetKV in 11 and InfoFlow KV in 12. Our W3 response reports service results across loads and deadlines.

In the submitted revision, we have added the matched selection comparison with ProphetKV and InfoFlow KV to the final paragraph of Section 4.4. Tables A9 and A10 in Appendix B.1 report quality and initialization timings at identical storage bytes and repair budgets. The second paragraph of Appendix C.5 specifies that both selection rules actually drive shared INT4 bases and private replacement overlays. The paragraph immediately after Table A19 reports their service and budget results.

AuthorsOpenReview ↗

W3 — Load, output length and deadline sensitivity under a total device budget

Thank you for suggesting tests across arrival rates, deadlines and output lengths.

The original 10.4% to 13.9% for OneBaseKV and 2.6% to 2.8% for Fusion-style are fractions of completed requests meeting 600 ms, not relative speedups. The original workloads test service under capacity pressure. We vary arrival load, TTFT deadline and output length under a 24 GiB whole-device budget, with controlled Fusion-style as the primary reference and matched query-aware methods as additional comparisons.

On one RTX 5090, NF4 Qwen3-8B uses 12K contexts. Per seed, each method replays the same 192 timestamped arrivals over 32 documents and 64 sessions. NVML peaks include weights, KV and temporary workspace. TTFT includes queueing from arrival. Outputs increase from 32 to 128 tokens. Arrival rate measures offered requests per second.

Below are three-seed mean window goodputs in req/s. Requests must finish in-window and meet the TTFT limit. † marks at least one run above 24 GiB. Bold marks the higher rate in each pair.

Output tokensArrivals/sFusion-style, 600 msOneBaseKV, 600 msFusion-style, 3000 msOneBaseKV, 3000 ms
320.03710.0010460.0050450.0081250.032689
320.11140.0009710.0114630.0102850.070361
320.18570.0003240.0068290.0013110.058269
1280.03710.000719†0.0039320.005767†0.023797
1280.11140.0001930.0005840.0011650.002722

At 32 tokens and 600 ms, OneBaseKV retains higher goodput than Fusion-style as arrivals increase across all three rates. With four times longer output, it remains higher at both rates and deadlines. At 128 output tokens and 0.1114 arrivals/s, the overall window completion rates are 0.08999 requests/s for OneBaseKV and 0.08493 for Fusion-style, counting all in-window completions regardless of TTFT.

OneBaseKV fits 24 GiB in all 15 runs and Fusion-style in 14. ProphetKV and InfoFlow KV exceed it in four and three runs, respectively. For each load, we compare only methods that remain within 24 GiB in all three seeds at that load. Under this load-specific comparison, OneBaseKV has the highest goodput at four of five loads under both deadlines, including both 128-token loads.

The eight-deadline open-arrival sweep below averages the within-seed OneBaseKV/Fusion-style goodput ratios. Both fit 24 GiB in every seed at these two 32-token loads. These statistics are separate from the original serial-trace deadline sweep.

TTFT ms0.0371 arrivals/s0.1114 arrivals/s
3004.87×12.50×
4504.87×12.67×
6004.87×12.67×
9004.48×13.17×
120016.34×41.00×
18002.61×4.98×
30004.56×8.37×
60001.28×2.82×

OneBaseKV has higher goodput at all eight deadlines and both arrival rates.

For absolute attainment at 32 tokens and 0.1114 arrivals/s, we follow all 576 arrivals through a further 120 s drain. Counts meeting 600 ms are OneBaseKV 60, InfoFlow KV 58 and Fusion-style five. ProphetKV exceeds 24 GiB in at least one run at this load. These counts include drain completions and are separate from window goodput.

At 32 tokens and 0.0371 arrivals/s, all three methods below fit 24 GiB in every seed. Entries give qualifying completions through the window and 120 s drain, and percentages. The denominator is all 576 arrivals.

TTFT msFusion-styleProphetKVOneBaseKV
60018, 3.13%78, 13.54%79, 13.72%
3000126, 21.88%497, 86.28%503, 87.33%
6000443, 76.91%554, 96.18%555, 96.35%

At 32 tokens and 0.0371 arrivals/s, OneBaseKV serves 96.35% of all arrivals within the 6000 ms TTFT deadline and the stated completion cutoff, identifying a service operating point at the longer deadline.

The submitted revision adds whole-device-budget service results to Section 4.3's final paragraph. In Appendix C.5, Table A19 compares arrival rates, TTFT deadlines and 32- and 128-token outputs. Table A20(a) gives the eight-deadline sweep at two arrival rates and A20(b) gives absolute attainment. C.5's third paragraph and the attainment paragraphs before Table A20 distinguish overall window completion rates, window goodput and attainment among all arrivals.

AuthorsOpenReview ↗

Q1 — Quality and latency when only the selector changes

We appreciate your suggestion to isolate selector effects using the same cache representation, repair budget and execution engine.

We compared five selection rules under the requested controls. Each model uses the same 100 HotpotQA questions and 100 Qasper questions for every rule. All methods use the same shared INT4 base, private INT4 replacement format, 30% repair budget and attention executor. Persistent bytes match per example. The model weights in this quality experiment are BF16. The KV representation being compared is INT4 for every selector.

The table reports macro quality across the four model and task combinations. The intervals are for each selector's score.

SelectorMacro quality (points)95% confidence interval
OneBaseKV50.94[46.17, 55.75]
FusionRAG47.53[42.61, 52.41]
ProphetKV51.86[46.97, 56.73]
InfoFlow KV50.40[45.55, 55.22]
CacheBlend49.82[44.99, 54.64]

For the paired comparison, we resample documents within tasks 10,000 times and keep methods paired on the same examples. OneBaseKV gains 3.41 points over FusionRAG selection, with a 95% interval of [1.41, 5.45]. Against ProphetKV the difference is −0.92 points with a 95% interval of [−2.42, 0.61]. Against InfoFlow KV it is +0.54 points with a 95% interval of [−0.84, 1.94]. Both of the latter paired intervals include zero, so these data do not resolve a directional quality difference with either selector.

The following table separates median probe time from full initialization time, both in milliseconds. Full initialization includes the probe, repair and packing.

Model and taskStageOneBaseKVFusionRAGProphetKVInfoFlow KVCacheBlend
Qwen3-8B, HotpotQAProbe54.467.282.060.92216.7
Qwen3-8B, HotpotQAFull initialization948.5960.0978.7956.33116.7
Qwen3-8B, QasperProbe38.142.159.442.3625.6
Qwen3-8B, QasperFull initialization330.9345.6363.5345.4925.8
Llama-3.1-8B, HotpotQAProbe46.860.267.252.12541.4
Llama-3.1-8B, HotpotQAFull initialization855.7867.4877.6862.03355.6
Llama-3.1-8B, QasperProbe33.436.751.538.7720.8
Llama-3.1-8B, QasperFull initialization296.7289.2311.2309.4980.8

Among all five rules, OneBaseKV has the lowest median probe time in all four combinations and the lowest median full initialization time in three. Across these four settings its median full initialization time is 64.3% to 74.5% lower than CacheBlend's, and its 95th-percentile full initialization time is 2.3% to 5.8% lower than ProphetKV's. The common representation and executor preserve distinct selector effects. OneBaseKV has higher quality than FusionRAG selection and lower probe and full initialization medians than ProphetKV and InfoFlow KV throughout the comparison.

In the submitted revision, we have added the quality comparison of five selection rules under a common cache representation, repair budget and executor to the final paragraph of Section 4.4. Table A9 in Appendix B.1 reports paired quality-difference confidence intervals, and Table A10 provides probe and full initialization timings for each model and task. The second paragraph of Appendix C.3 clarifies the distinction between model-weight and KV-storage precision.

Reviewer_BT2EOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the redundancy in KV caches that arises when documents are reused across multiple requests, this paper proposes a cache representation that combines a shared, low-bit base model with request-specific repair layers. By employing a unified, normalized form of compositional attention, the approach eliminates the need to persist full request-specific caches. The authors provide an equivalence analysis for this compositional execution and demonstrate its benefits in terms of both output quality and memory efficiency across various models, long-context tasks, and service workloads.

Critical strengths
  1. The approach is centered on the distinction between shared state and request-private state, with a seamless integration between the representation scheme and the execution mechanism. Table 5 empirically validates the roles of repair, quantization, sharing, and direct execution, providing strong support for the system design.
  2. The experiments encompass a variety of models and tasks, demonstrating that it is possible to reduce persistent KV-cache memory usage while maintaining performance levels comparable to those of a full cache. Furthermore, the quantitative comparison between the proposed combined execution and the corresponding materialized execution enhances the credibility of the results.
Critical weaknesses
  1. The 6.06x to 8.57x throughput gains highlighted in the abstract refer specifically to throughput achieved while meeting quality-of-service (QoS) requirements under defined thresholds, rather than an overall increase in throughput. As shown in Table A13, the proportion of requests meeting these requirements remains low; furthermore, the sensitivity of the current results to specific thresholds and workloads is not yet clearly established.
  2. The statistics for persistent KV caches exclude significant temporary workspace allocations, and some baselines utilize adapted implementations based on the specifications provided in the original papers. Consequently, the current results are insufficient to directly infer the actual gains in GPU memory efficiency and service capacity for a full-scale deployment.
Questions for authors
  1. Within the two existing sets of service traces, does the method’s throughput advantage over the baselines remain stable when the threshold for time-to-first-token (TTFT) is adjusted?
  2. Could you clarify the key differences between the adapted baseline implementations and their original, native counterparts? Additionally, please report the peak GPU memory usage per device, measured using a consistent methodology.
3 comments
AuthorsOpenReview ↗

W1 and Q1. Goodput, overall throughput and deadline sensitivity

Thank you for suggesting we distinguish overall throughput from SLO goodput and evaluate TTFT sensitivity.

The abstract's SLO goodput counts completions/s meeting the 600 ms TTFT limit. Overall throughput counts all completions. Original replays use a common 6 GiB persistent KV pool. Fusion-style denotes the shared-BF16 control. Qualifying percentages use each method's own completions as denominator. See Tables 4 and A16 in the revised paper.

Docs/sessionsMethodCompletions/s600 ms goodputQualifying %
16/48Fusion-style0.222120.006202.8%
16/48OneBaseKV0.361550.0514413.9%
32/64Fusion-style0.211510.005512.6%
32/64OneBaseKV0.319630.0336810.4%

Rates and shares are three-seed means. Ratios average within-seed comparisons, not quotients of mean rates. At each of eight TTFT limits, goodput divides qualifying completions by each run's original duration. Rates are requests/s. Grouped limits have identical results. Blank workload cells inherit the preceding value.

The original replays use BF16 model weights and compare OneBaseKV against Fusion-style and CacheBlend-style + INT4, with the latter storing a complete private KV cache per request. The last columns give OneBaseKV-to-control ratios.

Docs/sessionsTTFT msFusion-styleCacheBlend-style + INT4OneBaseKVvs Fusion-stylevs CacheBlend
16/48300,450,600,9000.006200.021420.051448.57×2.33×
12000.006200.021420.2675750.04×13.17×
18000.075850.021420.267573.59×13.17×
30000.075850.021420.361554.97×18.42×
60000.222120.170240.361551.63×2.12×
32/64300,450,600,9000.005510.018230.033686.06×1.82×
12000.005510.018230.1856733.57×10.38×
18000.062510.018230.185673.05×10.38×
30000.062510.018230.319635.39×18.24×
60000.211510.168790.319631.51×1.89×

Separate NF4 Qwen3-8B replays on one RTX 5090 use a 6 GiB pool. The INT4 control stores full private KV per request. Rates and paired ratios are computed separately.

Docs/sessionsTTFT msFull per-request INT4OneBaseKVPaired ratio
16/48300,450,600,9000.019310.034671.75×
1200,18000.019310.1816610.00×
30000.019310.2458014.01×
60000.153800.246401.60×
32/64300,450,600,9000.016920.023791.39×
1200,18000.016920.131477.94×
30000.016920.2267113.96×
60000.156860.226711.45×

OneBaseKV has higher goodput in every seed at all eight limits on both workloads in all three comparisons. At 1200 ms, the original replays admit 257 and 273 additional OneBaseKV completions across seeds on the smaller and larger workloads. Their TTFTs are about 949 to 986 ms, while Fusion-style's next request qualifies around 1215 ms. Thus the Fusion-style-relative ratio peaks at 1200 ms. At 6000 ms, all original completions qualify and ratios equal overall throughput ratios. Our W2 response tests arrival rates and output lengths under a total-device budget.

In the submitted revision, we have added overall completion rates to the final paragraph of Section 4.3 and the original traces' eight-limit comparison against Fusion-style to Table A17 in Appendix C.4. The paragraph immediately before Table A17 defines the paired statistics. The first paragraph after it explains the relative-goodput peak at 1200 ms.

AuthorsOpenReview ↗

W2. Service capacity under a total GPU memory budget

We appreciate the request to evaluate service capacity under a total GPU memory budget.

W2. The 37.7% figure measures persistent KV relative to BF16-COW at 12K and fanout 16, as shown in Figure 2(a) of the paper. To include weights and temporary workspace, we separately evaluate NF4 Qwen3-8B on one RTX 5090 with a 24 GiB total-device budget. NVML measures whole-device peak memory. For each seed, all methods replay the same 192 timestamped arrivals across 32 documents and 64 sessions. TTFT starts at arrival and includes queueing. The controlled Fusion-style implementation uses the same online selection rule and attention backend.

The table covers all five loads. The last two columns give three-seed mean goodput in requests/s. A request must finish within the measurement window and receive its first token within 600 ms. A dagger marks at least one over-budget seed at that load. Bold marks the higher goodput of the two methods.

Output tokensArrivals/sFusion-styleOneBaseKV
320.03710.0010460.005045
320.11140.0009710.011463
320.18570.0003240.006829
1280.03710.000719†0.003932
1280.11140.0001930.000584

OneBaseKV has higher goodput than Fusion-style at all five loads and meets the device budget in all 15 runs, compared with 14 for Fusion-style. At 0.1114 arrivals/s and 32 output tokens, both methods meet the budget in every seed. Their mean window goodputs are 0.011463 and 0.000971 requests/s, respectively. Computing OneBaseKV-to-Fusion-style ratios within seed and then averaging gives 12.67×. This comparison quantifies service benefits under a budget that includes weights, persistent state and temporary workspace. Our Q2 response compares whole-device peaks and service rates against full per-request INT4.

In the submitted revision, we have added service results across arrival rates and output lengths under the 24 GiB device budget to Table A19 in Appendix C.5. The first paragraph of C.5 specifies memory accounting, the third defines window goodput, and the paragraph immediately after Table A19 reports budget compliance.

AuthorsOpenReview ↗

Q2. Baseline implementations and measured device peaks

Thank you for asking us to clarify the baseline adaptations and report consistently measured device memory peaks.

Q2. Native FusionRAG builds chunk KV in retrieved-neighbor context and uses a specialized sparse attention operator. In the controlled serial replays, Fusion-style and OneBaseKV use the same frozen per-session repair positions and attention backend. During each session build, probing is rerun online and charged to TTFT, while repair uses the fixed positions. This compares storage on a common execution path. Full INT4 denotes the full per-request INT4 control, which selects positions online using a selector implemented to FusionRAG's specifications and stores a complete private KV cache per request. In the open-arrival experiments in our W2 response, online-selected positions determine repair.

The controlled implementation of CacheBlend retains KV-deviation selection and causal recomputation. At layer index 1, it compares full-prefill K/V with independently computed chunk KV, selects positions by mean absolute deviation and applies causal partial prefill. In the matched-selector experiment, both methods use the same shared INT4 base, private INT4 overlay, 30% repair budget and attention executor. Only selection varies. These are controlled adaptations, not unmodified native serving systems.

NVML measures whole-device peaks on one RTX 5090, including NF4 Qwen3-8B weights, KV and runtime allocations. Below are three-seed means. The first two workloads use a 6 GiB persistent KV pool. The third keeps the same two documents and four sessions resident. All runs meet the 24 GiB device budget. Lower peaks and higher rates are better. Blank workload cells inherit the preceding label.

Docs/sessionsMetricFusion-styleFull INT4OneBaseKV
16/48Peak GiB19.2818.0419.93
Completions/s0.17510.15380.2464
600 ms goodput req/s0.00490.01930.0347
32/64Peak GiB19.3119.6020.18
Completions/s0.16750.15690.2267
600 ms goodput req/s0.00440.01690.0238
2/4 residentPeak GiB16.7015.5813.82
Completions/s0.40790.20720.3378
600 ms goodput req/s0.27190.13810.2252

For the 32-document replay, pool use and resident sessions are averaged over request records. Resident sessions hold private overlays in shared methods and complete private caches in full INT4.

32 docs, 64 sessionsFusion-styleFull INT4OneBaseKV
Mean pool use %75.592.993.1
Mean resident sessions2.318.818.54

OneBaseKV has the highest overall completion rate and 600 ms goodput on both capacity-constrained workloads. Ratios below are computed within seed, then averaged, so need not equal ratios of the mean rates above.

OneBaseKV benefit (docs/sessions)vs. Fusion-stylevs. full INT4
600 ms goodput ratio (16/48)7.42×1.75×
600 ms goodput ratio (32/64)5.41×1.39×
Completion-rate ratio (32/64)1.35×1.45×
Peak saved (2/4 resident), GiB2.91.8

At 32 documents, OneBaseKV uses 0.87 and 0.58 GiB more peak memory than the respective controls, achieving these service gains within the same device budget. Overall completion rates include online probing and overlay construction. It retains 3.70× as many private overlays as Fusion-style and completes more requests/s at pool use comparable to full INT4.

With the same two documents and four sessions resident, without evictions or rebuilds, OneBaseKV has the lowest device peak, showing memory savings for the same resident workload. Table A1 of the paper compares quality and persistent KV. OneBaseKV and Fusion-style with full per-request INT4 score 50.91 and 47.50 in macro quality. OneBaseKV uses 40.8% less persistent KV at 12K and fanout 16.

In the submitted revision, we have added device peaks and service results for the two capacity-constrained workloads to Table A18(a), pool occupancy and resident-session counts to Table A18(b), and device peaks and service results for the fully resident workload to Table A18(c), all in Appendix C.4. The first two paragraphs after Table A18 explain these comparisons.

Reviewer_jRSROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes OneBaseKV, a low-bit KV-cache sharing framework for high-fanout retrieval-augmented generation (RAG). It aims to reduce redundant KV-cache storage across requests while preserving generation quality.

The main contributions are threefold. First, OneBaseKV stores each reusable document chunk as a single immutable INT4 canonical KV base shared across requests. Second, it introduces a query-aware repair mechanism that selectively recomputes query-sensitive KV positions and stores them in private INT4 replacement overlays. Third, it proposes a composite attention mechanism that combines shared canonical KV, private replacement KV, and dynamic BF16 KV through globally normalized online softmax, avoiding the materialization of a full per-request KV cache. The paper also provides theoretical analyses of attention equivalence and mixed-precision quantization errors.

The authors evaluate OneBaseKV on four LLMs and 13 LongBench tasks, with additional memory measurements, ablation studies, and serving experiments. The results show that OneBaseKV retains 99.10% of full-BF16 quality while reducing persistent KV storage to 37.7% of the BF16 copy-on-write baseline at a fanout of 16. Under a fixed 6-GiB persistent-KV pool, it achieves 6.06×–8.57× higher SLO goodput than Fusion-style.

Critical strengths
  1. Important and practical problem.

The paper addresses the redundant KV-cache storage problem in high-fanout RAG workloads, where multiple requests repeatedly access the same document chunks. This is a practically relevant problem for memory-efficient LLM serving, particularly under limited GPU memory capacity.

  1. Unified and well-motivated system design.

OneBaseKV integrates low-bit KV quantization, cross-request KV sharing, query-aware repair, and composite attention into a unified framework. The separation between shared canonical KV states and request-private replacement overlays is a reasonable design that reduces redundant storage while accommodating request-specific context differences. The proposed composite attention mechanism further avoids materializing a complete per-request KV cache.

  1. Comprehensive experimental evaluation with promising results.

The paper evaluates OneBaseKV across four LLMs and 13 LongBench tasks, together with memory measurements, controlled ablations, and serving experiments. The results demonstrate a favorable quality-memory tradeoff, retaining 99.10% of full-BF16 quality while using only 37.7% of BF16-COW persistent KV storage at fanout 16. The reported 6.06×–8.57× SLO-goodput improvements under constrained memory budgets further demonstrate the potential practical benefits of the proposed approach.

Critical weaknesses
  1. Unclear computational efficiency of low-bit composite attention.

OneBaseKV stores canonical and replacement KV states in INT4 but performs attention over dequantized representations. This introduces additional dequantization, replacement indexing, and composite attention overheads. Although the proposed representation reduces KV memory consumption, it remains unclear whether the reduction in memory traffic outweighs these additional computational costs. The paper lacks a detailed runtime breakdown and kernel-level latency comparisons against conventional BF16 attention and INT4 KV-cache attention, making it difficult to assess the actual computational efficiency of the proposed execution mechanism.

  1. Limited generalizability of serving performance gains beyond memory-constrained settings.

The reported 6.06×–8.57× SLO-goodput improvements are obtained under a fixed 6,144-MiB persistent-KV pool, where cache evictions and full rebuilds may substantially affect serving performance. Table 4 also shows that simply introducing INT4 storage into Fusion-style yields significant goodput improvements, suggesting that memory-capacity savings account for an important portion of the observed gains. However, the paper does not sufficiently evaluate whether OneBaseKV maintains its performance advantages under memory-sufficient conditions or across different KV-cache capacities. A memory-budget sensitivity study, together with comparisons under matched cache-residency conditions, would help establish the generalizability of the reported serving improvements.

Questions for authors

Q1. How is INT4 dequantization implemented during composite attention? Is dequantization fused into the attention kernel, or does it require separate operations and temporary buffers? Could the authors provide kernel-level latency comparisons against conventional BF16 attention and INT4 KV-cache attention to quantify the associated overhead?

Q2. Does OneBaseKV maintain its serving performance advantages when GPU memory is sufficient to avoid frequent KV-cache evictions and rebuilds? Could the authors evaluate different persistent-KV pool sizes and report TTFT, TPOT, and throughput under memory-sufficient conditions?

Q3. Table 4 shows that adding INT4 storage to Fusion-style already provides substantial SLO-goodput improvements. Could the authors further disentangle the contributions of low-bit quantization, cross-request sharing, replacement overlays, and composite attention to the overall serving performance gains?

5 comments
AuthorsOpenReview ↗

W1 and Q1. Decode cost, INT4 implementation and service benefits

Thank you for asking us to clarify the implementation and computational cost of low-bit composite attention.

W1 and Q1. Dequantization is separate from the attention kernel in this implementation. OneBaseKV unpacks two INT4 codes per byte, applies group scales and offsets, and inserts private replacements at their logical positions. The active layer's temporary buffer also holds dynamic KV. The shared base is never overwritten, and no dense request cache persists.

We time Qwen3-8B with BF16 model weights on one RTX 5090 at 12K context with 64 dynamic tokens, one query token and batch size one. Each value is the median of 300 CUDA-event measurements. The full decode step includes the model call, cache updates and greedy token selection, excluding only initial cache preparation.

Single-request decode latencyMaterialized BF16Full per-request INT4OneBaseKV
Median full decode step in ms25.8135.1045.01

Under the same configuration, we separately time single-layer cache preparation. Preparation includes updates and concatenation, adds unpacking for INT4 and adds replacement indexing for OneBaseKV. All three paths use the same attention operator. The common SDPA takes 0.146 ms on prepared single-layer inputs. Isolated stage medians are not added to estimate complete path latency.

Single-layer stage in msMaterialized BF16Full per-request INT4OneBaseKV
Cache preparation0.1140.3200.465

The 9.91 ms difference from full per-request INT4 is for the whole decode step and cannot be assigned to dequantization alone. Separate storage and service measurements quantify the benefits of the shared representation.

Sharing eliminates repeated document KV copies, while layerwise execution avoids retaining a complete dense KV cache spanning all layers for each active session. The first two rows below measure persistent KV at 12K and fanout 16, as reported in Table A7(b) of the paper. The last measures temporary allocation above resident state for 16 sessions decoding in rotation at 12K on an A100.

Memory measured in GiBComparison methodComparison valueOneBaseKVReduction
Persistent KVBF16-COW15.3055.77462.3%
Persistent KVFull per-request INT49.7565.77440.8%
Temporary decode allocationMaterialized execution27.330.3726.96 GiB

OneBaseKV and full per-request INT4 use 37.7% and 63.7% of BF16-COW persistent KV, respectively. OneBaseKV retains 99.10% of Full BF16 macro-average quality over 13 tasks. See Figure 2(a) and Table 2 of the paper.

Fusion-style denotes the shared-BF16 control. We also compare Fusion-style, full per-request INT4 and OneBaseKV on the same 32-document, 64-session replay with a 6 GiB persistent KV pool. This separate experiment uses Qwen3-8B with NF4 model weights on one RTX 5090. Time to first token includes online probing and overlay construction. The rates below are three-seed means.

Service metric in requests/sFusion-styleFull per-request INT4OneBaseKV
All completed requests0.16750.15690.2267
Completed requests with TTFT at most 600 ms0.00440.01690.0238

Including online work, OneBaseKV completes 1.35× as many requests per second as Fusion-style and 1.45× as many as full per-request INT4. Its 600 ms goodput is 5.41× and 1.39× that of these controls, respectively. Each ratio is calculated within seed and then averaged.

Packed KV storage introduces unpacking and input-preparation work in this implementation. The independent measurements show 40.8% less persistent KV than full per-request INT4, 26.96 GiB less temporary allocation than materialization with 16 active sessions, and higher actual completion rates in the fixed-pool replay. These memory and service benefits support the latency trade-off in the target capacity-constrained workload.

In the submitted revision, we have added the transient-memory comparison between layerwise and materialized execution with 16 active sessions to Table A11 and the second paragraph after it in Appendix B.1. The fixed-capacity service results appear in Table A18(a) and the first paragraph after Table A18 in Appendix C.4.

AuthorsOpenReview ↗

W2 and Q2. Cache capacity and service with fully resident sessions

We appreciate your suggestion to examine service performance across cache capacities and under full residency.

W2 and Q2. We vary persistent KV capacity and separately hold all sessions resident to remove eviction and rebuild effects. On one RTX 5090 with NF4 Qwen3-8B and a 6 GiB persistent KV pool, all three methods fit the 24 GiB device budget on both workloads in every seed. The first three numeric columns report three-seed mean completions/s meeting a 600 ms TTFT deadline. The last two give OneBaseKV-to-baseline ratios calculated within seed and then averaged.

Documents/sessionsOneBaseKVFusion-styleFull per-request INT4OneBaseKV / Fusion-styleOneBaseKV / per-request INT4
16 / 480.03470.00490.01937.42×1.75×
32 / 640.02380.00440.01695.41×1.39×

On the 32-document workload, increasing the pool from 6 to 12 GiB raises OneBaseKV's overall completion rate from 0.227 to 0.259 requests/s. At 12 GiB, it retains 11.48 document bases on average versus 4.80 for Fusion-style. The counts below sum full rebuilds over three seeds, with the same 576 requests per method.

Method at 12 GiBFull rebuilds across 576 requests
Fusion-style227
Full per-request INT4265
OneBaseKV99

OneBaseKV needs 56.4% fewer full rebuilds than Fusion-style and 62.6% fewer than full per-request INT4. When a document base remains resident, only the evicted session overlay needs rebuilding. The 12 GiB limit applies to persistent KV, not total device memory.

We then hold the same two documents and four sessions resident on the RTX 5090. All methods complete the replay without evictions or rebuilds. Rates and whole-device peaks are three-seed means. TTFT and time per output token are means of per-run medians.

MethodTTFT in msTPOT in msCompletion rate in req/sDevice peak in GiB
Fusion-style82.951.80.407916.70
Full per-request INT4140.2104.80.207215.58
OneBaseKV98.069.40.337813.82

For these identical resident sessions, OneBaseKV uses the least device memory, 2.9 GiB less than Fusion-style and 1.8 GiB less than full per-request INT4. Relative to full per-request INT4, it reduces TTFT by about 30% and TPOT by about 34% and completes 1.63× as many requests per second. Relative to Fusion-style, the additional latency accompanies a lower memory requirement for the same resident workload.

We also test full residency of the larger 32-document, 64-session workload on an A100-80GB. Both INT4 methods remain resident without eviction in all three runs. Fusion-style runs out of memory during base construction. OneBaseKV uses 25.88 GiB of peak persistent KV versus 40.50 GiB for full per-request INT4 and delivers 1.36× its completion rate. These A100 figures measure persistent KV, not the whole-device peaks in the RTX 5090 table.

The capacity-limited 32-document replay additionally gives OneBaseKV 1.35× the overall completion rate of Fusion-style and 1.45× that of full per-request INT4 with a 6 GiB pool, including online probing and overlay construction. Together, the controls quantify memory savings without evictions and increased service capacity when cache capacity is limited.

In the submitted revision, we have added the persistent-KV pool-capacity sweep to Table A18(d) and its discussion in Appendix C.4. Table A18(c) and its discussion compare latency and throughput with the same sessions fully resident. The final paragraph of C.4 gives the larger-working-set full-residency results on the A100-80GB. These passages distinguish persistent-KV capacity from peak device memory.

AuthorsOpenReview ↗

Q3. Separating quantization, sharing, repair and execution

Thank you for the opportunity to clarify the contributions of quantization, sharing, repair and execution.

Q3. We use controlled comparisons to separate low-bit representation, cross-request sharing, private repair and direct execution. The precision control fixes sharing and repair positions and changes only the base and overlay precision. The ownership control fixes INT4 precision and repair positions and changes whether each session stores a base copy. The repair control compares task quality with and without the private overlay. The execution control fixes the logical KV state and measures transient memory with direct or materialized execution. Table 5 of the paper reports repair quality. The additional service and transient-memory comparisons follow.

MechanismControl configurationCondition and measurementControlOneBaseKV
Low-bit representationShared BF16 base and BF16 overlay32 documents, 6 GiB pool, 600 ms goodput in req/s0.00450.0214
Cross-request sharingINT4 base replicated per sessionSame replay, 600 ms goodput in req/s0.01790.0214
Cross-request sharingINT4 base replicated per session2 documents and 4 resident sessions, peak persistent KV in MiB33712075
Private repairNo private overlaySeparate quality ablation, macro score30.9652.46
Direct executionMaterialized request cache12K and 16 interleaved sessions, transient allocation in GiB27.330.37

The first two rows use the same OneBaseKV runs on an A100. Keeping sharing and repair positions fixed, changing the base and overlay from BF16 to INT4 yields 4.77 times the control's 600 ms goodput and 1.19 times its overall completion rate. This comparison measures the service benefit of low-bit representation under the same sharing structure.

When both configurations already use INT4, sharing the base instead of replicating it raises goodput from 0.0179 to 0.0214 req/s. With the same two documents and four sessions fully resident, sharing reduces peak persistent KV from 3371 to 2075 MiB, saving 1296 MiB. Precision and repair positions remain fixed, so this byte reduction comes from removing per-session copies of the public base.

Private repair performs online probing, context-conditioned recomputation and overlay packing to recover request-specific KV representations. In the independent quality ablation reported in Table 5 of the paper, retaining the overlay raises macro quality from 30.96 to 52.46, recovering 21.50 points. This quantifies the quality obtained from the additional request-specific computation.

Materialized execution first constructs a complete dequantized request KV cache and uses it for decoding in the serial replay. OneBaseKV's direct execution processes the same logical KV state layer by layer without retaining a complete dense cache for each session. In the separate 12K experiment with 16 sessions decoding in rotation, transient allocation above resident state is 27.33 GiB for the materialized control and 0.37 GiB for OneBaseKV, saving 26.96 GiB. Across five service events, both paths produce identical 32-token sequences.

These controls quantify the service gains from low-bit representation and sharing, the quality recovered by repair and the transient-memory savings from direct execution under their respective experimental conditions. The effects are reported separately and are not added to attribute an overall throughput gain.

In the submitted revision, we have added the service and memory controls for quantization, base sharing and direct execution to Table A11 in Appendix B.1. The paragraph immediately before Table A11 defines the controls, and the two paragraphs immediately after it explain repair and execution. These passages connect the new controls to the repair-quality results already in Table 5 of the paper to explain each mechanism's role.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the submitted revision

We thank the reviewers for their questions and suggestions. We have uploaded a revised manuscript and addressed the questions through additional controlled experiments and clarifications. The additions and updates in the submitted revision and their locations are summarized below. The final paragraphs of Sections 4.3 and 4.4 summarize the new findings, with the detailed results in the appendices.

Appendix B.1 presents comparisons with FusionRAG, ProphetKV, InfoFlow KV and CacheBlend selection rules under identical shared INT4 storage, repair budgets and attention execution. Tables A9 and A10 report quality, paired confidence intervals and initialization timings. Table A11 and its surrounding discussion separate the service effects of precision and sharing, the quality contribution of repair and the transient-memory savings from direct execution. The paragraph immediately before Table A11 defines the controls, and the two paragraphs immediately after it explain repair and execution.

The second paragraph of Appendix C.3 clarifies model-weight and KV-storage precision. Appendix C.4 reports the original traces across eight TTFT thresholds in Table A17. Its immediately preceding paragraph defines the paired statistics, and the first paragraph after it explains the 1200 ms peak. Table A18(a) gives device peaks and service rates under capacity pressure, panel (b) gives pool occupancy and resident-session counts, panel (c) compares the same sessions fully resident, and panel (d) gives full rebuilds with a 12 GiB pool. The following discussion explains these controls, and the final paragraph of C.4 gives full residency of the larger working set on an A100-80GB. Appendix C.5 evaluates arrival rates, service deadlines and output lengths under a 24 GiB whole-device budget in Table A19. The first three paragraphs of Appendix C.5 specify the setup and window metrics, and the paragraph immediately after Table A19 reports budget compliance and the selector service comparison. Table A20(a) gives the eight-deadline sweep, while panel (b) gives absolute attainment. The attainment paragraphs before Table A20 define the all-arrival denominator and completion cutoff separately from window rates.

The additional results identify the conditions and mechanisms supporting OneBaseKV's quality, memory and service benefits. We provide detailed responses to each weakness and question under the corresponding reviews.

Thank you for the follow-up and for considering our responses and additional experiments. We appreciate your constructive feedback, which has helped us improve the clarity and experimental evaluation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.