Scalable Attention: The Inference Compute-Cache Scaling Law

We present Scalable Attention (SA), which scales attention compute without enlarging the KV cache, adding marginal parameters. Based on SA, we propose the compute-cache scaling law, which models loss as separate power laws in compute and cache. Given unit prefill and decode costs, set by prefix caching, workload, speculative decoding, and hardware, the law returns the optimal compute and cache for any target loss. At the 3.3B scale, SA with twice the attention compute outperforms GQA in validation loss while using only $25\%$ of its KV cache.