Exact Finite-Query KV Cache Compilation
Abstract
Query-aware key value (KV) cache compression fits a chosen slot count to reference queries, but a successful fit cannot show that fewer slots would fail. We characterize the number of shared KV slots needed to preserve the normalized attention outputs of a finite protected query set exactly. A ratio identity maps each query to a ray of feasible replacement values. With query-specific positive weights, the minimum count is one plus the smallest dimension of an affine subspace meeting all rays. Two queries can already rule out one slot. For grouped query attention (GQA) with query heads per KV head, protecting positions requires slots for entries under explicit rank and key-realizability conditions. All 431,256 evaluated KV-head configurations satisfy the conditional law. With full query rank and realizable weights, a related construction represents an entire prefix by standard KV slots per head for a known -step trace. Across four models, native 32-bit floating-point (FP32) execution preserves every protected token through eight steps, removing 89.31% to 96.95% of same-dtype KV tensor bytes on 524-slot prefixes. The results separate fitting a cache from certifying its capacity within the shared KV representation.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.