OneBaseKV: Low-Bit Canonical KV Sharing with Per-Request Replacement Overlays for High-Fanout RAG

Long-context services increasingly process recurring content across many requests. High-fanout retrieval-augmented generation (RAG) is a representative workload in which the same document chunks are repeatedly combined with different queries, orders, and retrieval results. Autoregressive inference retains key–value (KV) states for these long inputs, so per-request cache ownership replicates document state and rapidly consumes memory even after low-bit quantization. Exact prefix sharing removes duplication only for byte-identical prefixes, while cache-fusion systems do not encode shared document state and query-conditioned repairs as jointly executable low-bit objects. We propose OneBaseKV, which stores each reusable chunk once as an immutable INT4 canonical base and represents query-sensitive differences with private INT4 replacement overlays. An architecture-normalized selector identifies repair positions, and materialization-equivalent composite attention merges the base, overlay, and dynamic KV through one global online softmax without a persistent dense request cache. We prove exact equivalence to standard attention over the corresponding materialized dequantized KV and derive a mixed-partition perturbation bound for heterogeneous precisions. Experiments show that OneBaseKV achieves the best quality among the compared efficient methods under fixed persistent-KV budgets. At a $0.52\times$ BF16 copy-on-write (BF16-COW) persistent-KV cap, it exceeds an oracle choosing the best feasible external method for each combination of model and task by 5.48%. At a 12K-token context and fanout 16, OneBaseKV uses only 37.7% of BF16-COW persistent KV. In two fixed-capacity serving traces, it achieves $6.06\times$–$8.57\times$ the service-level objective (SLO) goodput of Fusion-style. On non-retrieval RULER at 32K, it exceeds every efficient external method on three of four models.