LCRA: Local Attention with Hierarchically Routed Coarse-Residual Memory for Long-Context Retrieval

Recurrent language models compress distant history into fixed-size states, but such compression can make individually retrievable associations difficult to preserve as new information accumulates. Existing gated or delta-rule memories primarily focus on how a shared recurrent state is updated, rather than explicitly partitioning memory across associations. Routing-based memories provide an organizational mechanism, but coarse aggregation within a route can still merge distinct values. We introduce LCRA, a Local-Coarse-Residual Architecture that combines fixed-window local FoX attention for recent context with hierarchically routed coarse and residual recurrent memory for distant context. Route-conditioned value-and-mass statistics provide coarse predictions, while grouped fast-weight states use residual targets relative to pre-write coarse estimates. The resulting remote readout is fused with the local branch, while the size of the remote recurrent state remains independent of context length. Across seven approximately 720M-parameter checkpoints, LCRA remains competitive on language modeling and general zero-shot tasks while showing its clearest gains in long-context retrieval. It leads all four evaluated 8K RULER-style synthetic retrieval tasks and retains 76.95% accuracy on 16K S-NIAH-1, compared with 19.53% for GDN-2. LCRA also achieves the lowest fixed-target PG-19 loss through 16K and the highest real-world in-context retrieval mean among the evaluated fixed-state models, while trailing the KV-access references on that mean. A separate 3B-token ablation study supports the complete remote-memory configuration and reveals task- and length-dependent trade-offs in the residual write target. Together, these results position LCRA as a promising fixed-state architecture for long-context retrieval.