- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Tayra extends Raven's sparse key-value recurrent memory with cumulative state snapshots and a hierarchical attention interface. Each snapshot is indexed using an anchor-query-weighted centroid of its stored keys and an entropy term. Text queries use these indices to weight historical readouts, while native queries attend within individual states. Online and historical outputs are jointly normalized, with a candidate-count correction applied to the online logit. The architecture combines recurrent-state archiving with a HiLS-style indexing and readout mechanism. Experiments train models on 50B tokens and evaluate general language ability, LongBench, in-context retrieval, and RULER up to 32K context length.
Critical strengths
-
The empirical gains over the Raven-based MARCH adaptation are meaningful. The models share training data and a 50B-token budget, have similar parameter counts, and use the same archival interval. MARCH also retains cumulative snapshots and reads history densely. Tayra improves LongBench from 13.1 to 16.1, winning on 10 of 12 tasks, and complex NIAH from 26.48 to 32.25 (Tables 2 and 4). These comparisons support the effectiveness of the complete architecture and are stronger evidence than a comparison with an unaugmented recurrent backbone alone.
-
The method is described clearly, including the distinction between routing queries and native reading queries. The KL identity in Eq. (12) is correct, and its interpretation is appropriately limited. Separately trained layer-placement ablations also provide a useful empirical observation: five middle layers nearly match all-layer NIAH performance and substantially outperform five interleaved layers.
Critical weaknesses
-
The methodological advance over the cited prior work is limited, and the experiments do not establish the independent value of the new adaptations. Raven already provides the sparse key-value recurrence; MARCH provides cumulative state snapshots and anchor-based access; HiLS provides the key-centroid/entropy index, hierarchical readout, and low-rank routing-query calibration (HiLS, Sections 3 and 4.1). Tayra's distinctive step is to apply this attention construction to compressed recurrent snapshots and introduce a candidate-count correction. This is a sensible integration, but the paper does not yet demonstrate a substantial conceptual advance beyond these ingredients. That concern is compounded by the missing component controls: Table 5 varies read placement, query parameterization, and retrieval budget, but does not isolate stored-key indexing, joint online/history fusion, or the log n_t correction. The MARCH comparison changes several of these choices together, as well as routing dimensions and head sharing. The independent-query variant still uses stored-key indices. Consequently, the reported gains validate the combined system without showing which proposed adaptation is responsible for the gains. This limited and insufficiently isolated contribution is my main reason for recommending weak rejection.
-
The practical benefit at a given resource budget remains unclear. Main results use dense historical reading in all 21 layers. Section 3.4 correctly describes the growing access and storage costs, but no throughput, latency, or peak-memory measurements are provided. The 256-slot snapshots are stored every 512 tokens in FP32, so a reduction in slot count alone does not establish a corresponding memory or speed advantage. Top-4 retrieval lowers mean NIAH from 59.57 to 51.29 while retaining the archive and scanning all indices. Without a measured quality-cost comparison, the empirical gains provide limited evidence for the practical advantage of this integration.
Questions for authors
-
Beyond transferring HiLS-style attention to recurrent snapshots, what specific methodological insight is supported by the experiments? In particular, can you provide a comparison that holds the backbone, archive, routing dimensions, read layers, and fusion rule fixed while changing only anchor-representation versus stored-key indexing? Evidence isolating joint fusion and the log n_t correction would also materially affect my assessment of the contribution.
-
Can you report prefill/decode throughput and peak memory or cache bytes at 16K and 32K for dense Tayra, Top-4 Tayra, and Raven-based MARCH under the same hardware, batch size, and precision settings? Corresponding Transformer measurements, if available, would help establish whether the accuracy gains translate into a favorable quality-cost trade-off.
3 comments
Thank you for your careful response. Regrettably, I will maintain my original score. Novelty remains my top reviewing priority. I had the idea of incorporating snapshots into linear-complexity models as early as spring 2025. While I entertained this concept then, I did not find it motivating enough to pursue. In my view, this level of novelty is insufficiently impactful. I remember a closely related work appearing at NeurIPS 2025. I group such approaches under the umbrella of RAG + linear models. The concept is simple, yet it comes with additional implementation overhead. I am eager to see more fundamental algorithmic innovations, for instance in the state update equations. Admittedly this area is extremely crowded, but I would welcome refreshing, insightful advances here.
On the evaluation budget: I encourage the authors to compare against standard, community-recognized SOTA models including GDN and KDA. Furthermore, Mamba3 and GDN2 have been released this year, and new related methods continue to appear. Comparisons against these newer front-running models would strengthen the paper.
We agree that Tayra can broadly be viewed as part of the RAG+linear-models paradigm. However, conventional RAG performs routing outside the language model, whereas Tayra studies native memory routing inside a recurrent architecture. Historical states, their indices, and memory reads are all integrated into the model’s forward computation.
Different focus from GDN, KDA, Mamba3, and GDN2.
These methods mainly improve how a compact single-state memory is updated. State-update design is an important but already highly crowded research direction. Tayra studies a different architectural question: how a model can preserve, route, and revisit information after it has been compressed. Its core idea is to make persistent historical memory addressable within the recurrent model itself, rather than relying on a single overwritten state. MARCH has already extensively evaluated the single-state update family, including GDN, KDA, and GDN2. Raven is therefore the structurally relevant baseline for Tayra and is also more recent than GDN2.
Why we believe Hybrid Architectures Are Not the Endpoint.
Hybrid linear–full-attention models are useful engineering compromises, but we believe that they remain fundamentally constrained by their full-attention layers. As agentic trajectories grow through repeated tool calls and interaction steps, token-level KV caches continue to grow, while the context remains difficult to consolidate into a compact, persistent memory. The Mistral Large 4 technical report provides a concrete example: an initial cache-saving design reduced memory substantially but degraded long-context agentic performance, leading to broader long-context access in the final design (technical report).
Toward Native Persistent-Memory Models.
Tayra aims to move beyond this compromise by making historical context a native, persistent, and addressable recurrent resource. In a long-running agent, the live recurrent state could track task progress across tool calls, while indexed snapshots could provide access to earlier evidence when it becomes relevant again. This vision aligns with the emphasis on actively maintaining task-relevant context across steps in Context Language Models and You Don’t Need Long Context. We believe this design could lay the foundation for a new family of models built for real-world, long-horizon agentic tasks. We envision scaling Tayra to larger models and exploring the new training paradigms this architecture could enable, including test-time training and continual learning. We will add this discussion and both references to the main text of the revised manuscript. We look forward to sharing our progress with the community.
Response to W1 and Q1 — Contributions and methodological insights
We agree that the methodological contribution should be explicit and supported by controlled comparisons.
While we build on Raven's dual-state recurrence and HiLS's index construction, Tayra is, to our knowledge, the first to derive cross-snapshot routing indices directly from the stored key slots of cumulative recurrent memories, sharing those keys between snapshot selection and native slot-level addressing. This extension is non-trivial because recurrent updates, compression, and forgetting change the key structure retained in each snapshot. Each index is constructed from the retained SK slots, which also determine the within-snapshot attention weights used to read SV. Historical selection is therefore grounded in the addressing structure available to the native reader. We support this design through extensive experiments, including controlled component ablations and interventions on historical access across depth.
We further identify a strong dependence of historical-memory use on depth. Using the original C=512 dense checkpoint, we disable historical reads in different seven-layer groups on 600 paired S2/S3 examples across 4K, 8K, and 16K, holding model weights and the unmodified body-prefix cache fixed.
Changes are relative to all-layer reading. Removing middle-layer reads causes the largest degradation, at 39.67%. In separately trained C=256 models, five contiguous middle read layers achieve comparable mean NIAH6 accuracy to all 21 layers, at 64.43% and 64.07%, respectively, averaged over 4K, 8K, and 16K. These results provide an empirical basis for concentrating historical retrieval in intermediate layers.
Response to W1 and Q1 — Controlled component ablations
All ablations use the same backbone architecture. Each variant changes only the indicated component. Mean NIAH averages six tasks at 4K, 8K, 16K, and 32K.
Stored-key indexing. The anchor-derived variant retains entropy correction and joint fusion, directly testing the routing-key representation under a fixed fusion rule. Accuracy decreases by 6.85%, supporting indices derived from the key structure retained within recurrent snapshots.
Joint online/history fusion. Replacing joint normalization with MARCH-style additive fusion while retaining stored-key indexing reduces accuracy by 19.73 points, providing evidence for joint normalization within the same indexing configuration.
Candidate-count correction and entropy. Removing lognt or the entropy term reduces accuracy by 24.97 and 20.27 points, respectively. These matched comparisons support the contributions of both calibration terms to long-context recall and address the attribution concern about the original combined-system comparison.
Response to W2 and Q2 — Practical efficiency and the quality–cost trade-off
We agree that slot count should be accompanied by measurements of execution costs and storage requirements. Historical key/value snapshots are stored in BF16, using two bytes per element. We will clarify this storage precision in the revised paper.
We benchmark all five configurations at 16K and 32K on one GPU, using batch size 4, generation length 1,024 tokens, and BF16 operator precision. TTFT includes prompt processing and the first output token. Timings include historical archive updates. Tayra retains 256 slots per snapshot, archived every 512 tokens.
Each context-length cell reports TTFT (ms)/decode throughput (tokens/s)/analytical cache (GiB). Mean NIAH averages six tasks at 4K, 8K, and 16k.
Dense Tayra improves mean NIAH over MARCH by 7.18%, with lower TTFT and higher decode throughput at both lengths, using approximately 5% larger analytical cache. Top-4 trades an 8.28-point accuracy reduction for 1.44x and 1.71x decode throughput relative to dense Tayra at 16K and 32K. These measurements quantify the speed–accuracy trade-off alongside storage requirements.