Tayra: Hierarchical Attention over Evolving Recurrent States

Recurrent state space models (SSMs) enable efficient context processing by compressing past inputs into compact states, but continued updates can make earlier information harder to recover. Larger state spaces and multiple-state architectures extend recurrent memory, yet its usefulness also depends on how queries locate and read the information retained within these states. We introduce **Tayra**, a recurrent memory architecture in which the keys used to read a state also provide the basis for addressing it. Tayra combines a sparsely updated online memory with archived key–value states. Compact indices derived from stored keys guide historical access without replacing the states' internal contents, which remain available for query-dependent reading. A joint attention interface combines online and historical readouts. After pretraining on 50B tokens, Tayra achieves a 43.4% relative improvement over Raven in mean accuracy on six needle-in-a-haystack tasks across 4K-16K contexts. It also improves the average LongBench score by approximately 23% over a Raven-based MARCH adaptation. Recall gains extend to twice the training context length, while general language capabilities are preserved.