MANTA: Linearizing Large Language Models with Multi-Center Taylor Attention

Softmax attention gives large language models flexible access to the preceding context, but its quadratic computation and growing key–value cache limit efficient long-context deployment. Converting a pretrained softmax Transformer into a fixed-state model offers a practical alternative to pretraining a new architecture from scratch. However, Taylor-based linearization typically expands all keys around one global center. This approximation is inaccurate when pretrained keys form broad, anisotropic distributions. We introduce MANTA (Multi-Center Linear Attention via Taylor Approximation), which uses query-aware routing to make the softmax approximation local. MANTA maintains an additive recurrent state for each routed group, while the total state size remains independent of context length. It projects the memory-dominant matrix in each state onto a learned low-rank basis to reduce memory. Finally, joint normalization combines the compressed history with sliding-window attention. Experiments show that MANTA recovers the language-modeling quality of the original softmax model, with an average score of $73.2$ across six benchmarks. Compared with the softmax baseline, MANTA improves decoding efficiency and achieves up to $27.3\times$ the throughput. By making the Taylor approximation local and memory-efficient, MANTA provides a practical path from pretrained softmax Transformers to accurate fixed-state models with linear total computation and memory independent of context length.