acceptodds
Under review as a conference paper at ICLR 2027

Tayra: Hierarchical Attention over Evolving Recurrent States

Abstract

Recurrent state space models (SSMs) enable efficient context processing by compressing past inputs into compact states, but continued updates can make earlier information harder to recover. Larger state spaces and multiple-state architectures extend recurrent memory, yet its usefulness also depends on how queries locate and read the information retained within these states. We introduce **Tayra**, a recurrent memory architecture in which the keys used to read a state also provide the basis for addressing it. Tayra combines a sparsely updated online memory with archived key–value states. Compact indices derived from stored keys guide historical access without replacing the states' internal contents, which remain available for query-dependent reading. A joint attention interface combines online and historical readouts. After pretraining on 50B tokens, Tayra achieves a 43.4% relative improvement over Raven in mean accuracy on six needle-in-a-haystack tasks across 4K-16K contexts. It also improves the average LongBench score by approximately 23% over a Raven-based MARCH adaptation. Recall gains extend to twice the training context length, while general language capabilities are preserved.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, State Space Models, Recurrent Memory, Long-Context Modeling, Memory Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YYRKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Tayra extends Raven's sparse key-value recurrent memory with cumulative state snapshots and a hierarchical attention interface. Each snapshot is indexed using an anchor-query-weighted centroid of its stored keys and an entropy term. Text queries use these indices to weight historical readouts, while native queries attend within individual states. Online and historical outputs are jointly normalized, with a candidate-count correction applied to the online logit. The architecture combines recurrent-state archiving with a HiLS-style indexing and readout mechanism. Experiments train models on 50B tokens and evaluate general language ability, LongBench, in-context retrieval, and RULER up to 32K context length.

Critical strengths
  1. The empirical gains over the Raven-based MARCH adaptation are meaningful. The models share training data and a 50B-token budget, have similar parameter counts, and use the same archival interval. MARCH also retains cumulative snapshots and reads history densely. Tayra improves LongBench from 13.1 to 16.1, winning on 10 of 12 tasks, and complex NIAH from 26.48 to 32.25 (Tables 2 and 4). These comparisons support the effectiveness of the complete architecture and are stronger evidence than a comparison with an unaugmented recurrent backbone alone.

  2. The method is described clearly, including the distinction between routing queries and native reading queries. The KL identity in Eq. (12) is correct, and its interpretation is appropriately limited. Separately trained layer-placement ablations also provide a useful empirical observation: five middle layers nearly match all-layer NIAH performance and substantially outperform five interleaved layers.

Critical weaknesses
  1. The methodological advance over the cited prior work is limited, and the experiments do not establish the independent value of the new adaptations. Raven already provides the sparse key-value recurrence; MARCH provides cumulative state snapshots and anchor-based access; HiLS provides the key-centroid/entropy index, hierarchical readout, and low-rank routing-query calibration (HiLS, Sections 3 and 4.1). Tayra's distinctive step is to apply this attention construction to compressed recurrent snapshots and introduce a candidate-count correction. This is a sensible integration, but the paper does not yet demonstrate a substantial conceptual advance beyond these ingredients. That concern is compounded by the missing component controls: Table 5 varies read placement, query parameterization, and retrieval budget, but does not isolate stored-key indexing, joint online/history fusion, or the log n_t correction. The MARCH comparison changes several of these choices together, as well as routing dimensions and head sharing. The independent-query variant still uses stored-key indices. Consequently, the reported gains validate the combined system without showing which proposed adaptation is responsible for the gains. This limited and insufficiently isolated contribution is my main reason for recommending weak rejection.

  2. The practical benefit at a given resource budget remains unclear. Main results use dense historical reading in all 21 layers. Section 3.4 correctly describes the growing access and storage costs, but no throughput, latency, or peak-memory measurements are provided. The 256-slot snapshots are stored every 512 tokens in FP32, so a reduction in slot count alone does not establish a corresponding memory or speed advantage. Top-4 retrieval lowers mean NIAH from 59.57 to 51.29 while retaining the archive and scanning all indices. Without a measured quality-cost comparison, the empirical gains provide limited evidence for the practical advantage of this integration.

Questions for authors
  1. Beyond transferring HiLS-style attention to recurrent snapshots, what specific methodological insight is supported by the experiments? In particular, can you provide a comparison that holds the backbone, archive, routing dimensions, read layers, and fusion rule fixed while changing only anchor-representation versus stored-key indexing? Evidence isolating joint fusion and the log n_t correction would also materially affect my assessment of the contribution.

  2. Can you report prefill/decode throughput and peak memory or cache bytes at 16K and 32K for dense Tayra, Top-4 Tayra, and Raven-based MARCH under the same hardware, batch size, and precision settings? Corresponding Transformer measurements, if available, would help establish whether the accuracy gains translate into a favorable quality-cost trade-off.

3 comments

Response to W1 and Q1 — Contributions and methodological insights

We agree that the methodological contribution should be explicit and supported by controlled comparisons.

While we build on Raven's dual-state recurrence and HiLS's index construction, Tayra is, to our knowledge, the first to derive cross-snapshot routing indices directly from the stored key slots of cumulative recurrent memories, sharing those keys between snapshot selection and native slot-level addressing. This extension is non-trivial because recurrent updates, compression, and forgetting change the key structure retained in each snapshot. Each index is constructed from the retained SKS^K slots, which also determine the within-snapshot attention weights used to read SVS^V. Historical selection is therefore grounded in the addressing structure available to the native reader. We support this design through extensive experiments, including controlled component ablations and interventions on historical access across depth.

We further identify a strong dependence of historical-memory use on depth. Using the original C=512 dense checkpoint, we disable historical reads in different seven-layer groups on 600 paired S2/S3 examples across 4K, 8K, and 16K, holding model weights and the unmodified body-prefix cache fixed.

Historical-read settingAccuracy (%)Change (pp)Gold-answer ΔNLL
All 21 layers59.17——
Front seven disabled (0–6)58.00−1.17−0.0078
Middle seven disabled (7–13)19.50−39.671.6284
Back seven disabled (14–20)48.00−11.170.0375

Changes are relative to all-layer reading. Removing middle-layer reads causes the largest degradation, at 39.67%. In separately trained C=256 models, five contiguous middle read layers achieve comparable mean NIAH6 accuracy to all 21 layers, at 64.43% and 64.07%, respectively, averaged over 4K, 8K, and 16K. These results provide an empirical basis for concentrating historical retrieval in intermediate layers.

Response to W1 and Q1 — Controlled component ablations

All ablations use the same backbone architecture. Each variant changes only the indicated component. Mean NIAH averages six tasks at 4K, 8K, 16K, and 32K.

ConfigurationMean NIAH (%)
Full Tayra58.63
Replace stored-key indexing with anchor-derived indexing51.78
Replace joint normalization with MARCH-style additive fusion38.90
Remove candidate-count correction33.66
Remove entropy term38.36

Stored-key indexing. The anchor-derived variant retains entropy correction and joint fusion, directly testing the routing-key representation under a fixed fusion rule. Accuracy decreases by 6.85%, supporting indices derived from the key structure retained within recurrent snapshots.

Joint online/history fusion. Replacing joint normalization with MARCH-style additive fusion while retaining stored-key indexing reduces accuracy by 19.73 points, providing evidence for joint normalization within the same indexing configuration.

Candidate-count correction and entropy. Removing log⁡nt\log n_t or the entropy term reduces accuracy by 24.97 and 20.27 points, respectively. These matched comparisons support the contributions of both calibration terms to long-context recall and address the attribution concern about the original combined-system comparison.

Response to W2 and Q2 — Practical efficiency and the quality–cost trade-off

We agree that slot count should be accompanied by measurements of execution costs and storage requirements. Historical key/value snapshots are stored in BF16, using two bytes per element. We will clarify this storage precision in the revised paper.

We benchmark all five configurations at 16K and 32K on one GPU, using batch size 4, generation length 1,024 tokens, and BF16 operator precision. TTFT includes prompt processing and the first output token. Timings include historical archive updates. Tayra retains 256 slots per snapshot, archived every 512 tokens.

Each context-length cell reports TTFT (ms)/decode throughput (tokens/s)/analytical cache (GiB). Mean NIAH averages six tasks at 4K, 8K, and 16k.

ConfigurationMean NIAH16K32K
Transformer65.17558.7/608.2/8.8591727.2/426.0/17.719
Raven41.53510.7/1586.3/0.138994.8/1585.4/0.138
Raven-based MARCH52.39964.4/746.2/4.3572250.1/634.2/8.577
Dense Tayra, chunk 51259.57846.6/784.8/4.5742108.4/655.6/9.009
Top-4 Tayra, chunk 51251.29997.3/1128.2/4.5741921.4/1122.4/9.009

Dense Tayra improves mean NIAH over MARCH by 7.18%, with lower TTFT and higher decode throughput at both lengths, using approximately 5% larger analytical cache. Top-4 trades an 8.28-point accuracy reduction for 1.44x and 1.71x decode throughput relative to dense Tayra at 16K and 32K. These measurements quantify the speed–accuracy trade-off alongside storage requirements.

Thank you for your careful response. Regrettably, I will maintain my original score. Novelty remains my top reviewing priority. I had the idea of incorporating snapshots into linear-complexity models as early as spring 2025. While I entertained this concept then, I did not find it motivating enough to pursue. In my view, this level of novelty is insufficiently impactful. I remember a closely related work appearing at NeurIPS 2025. I group such approaches under the umbrella of RAG + linear models. The concept is simple, yet it comes with additional implementation overhead. I am eager to see more fundamental algorithmic innovations, for instance in the state update equations. Admittedly this area is extremely crowded, but I would welcome refreshing, insightful advances here.

On the evaluation budget: I encourage the authors to compare against standard, community-recognized SOTA models including GDN and KDA. Furthermore, Mamba3 and GDN2 have been released this year, and new related methods continue to appear. Comparisons against these newer front-running models would strengthen the paper.

We agree that Tayra can broadly be viewed as part of the RAG+linear-models paradigm. However, conventional RAG performs routing outside the language model, whereas Tayra studies native memory routing inside a recurrent architecture. Historical states, their indices, and memory reads are all integrated into the model’s forward computation.

Different focus from GDN, KDA, Mamba3, and GDN2.

These methods mainly improve how a compact single-state memory is updated. State-update design is an important but already highly crowded research direction. Tayra studies a different architectural question: how a model can preserve, route, and revisit information after it has been compressed. Its core idea is to make persistent historical memory addressable within the recurrent model itself, rather than relying on a single overwritten state. MARCH has already extensively evaluated the single-state update family, including GDN, KDA, and GDN2. Raven is therefore the structurally relevant baseline for Tayra and is also more recent than GDN2.

Why we believe Hybrid Architectures Are Not the Endpoint.

Hybrid linear–full-attention models are useful engineering compromises, but we believe that they remain fundamentally constrained by their full-attention layers. As agentic trajectories grow through repeated tool calls and interaction steps, token-level KV caches continue to grow, while the context remains difficult to consolidate into a compact, persistent memory. The Mistral Large 4 technical report provides a concrete example: an initial cache-saving design reduced memory substantially but degraded long-context agentic performance, leading to broader long-context access in the final design (technical report).

Toward Native Persistent-Memory Models.

Tayra aims to move beyond this compromise by making historical context a native, persistent, and addressable recurrent resource. In a long-running agent, the live recurrent state could track task progress across tool calls, while indexed snapshots could provide access to earlier evidence when it becomes relevant again. This vision aligns with the emphasis on actively maintaining task-relevant context across steps in Context Language Models and You Don’t Need Long Context. We believe this design could lay the foundation for a new family of models built for real-world, long-horizon agentic tasks. We envision scaling Tayra to larger models and exploring the new training paradigms this architecture could enable, including test-time training and continual learning. We will add this discussion and both references to the main text of the revised manuscript. We look forward to sharing our progress with the community.

Reviewer_TiFPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Tayra, a hierarchical recurrent memory architecture designed to improve long-context retrieval. It combines sparsely updated online memory with archived key–value states and introduces anchor-based indexing and a joint readout mechanism to access historical information. Experiments on general language benchmarks, LongBench, and RULER show improvements over recurrent baselines, particularly in long-context understanding and associative recall.

Critical strengths
  1. Tayra derives state-level indices directly from archived keys, allowing memory addressing and content retrieval to rely on the same stored representations. The theoretical analysis also establishes a connection between the indexing scores and the log-partition function, providing a mathematical basis for the proposed mechanism.
  2. The experiments cover general language understanding, long-context benchmarks, and associative recall tasks. Tayra achieves notable improvements over recurrent baselines, particularly on LongBench and RULER. Its performance at 32K also indicates an ability to generalize beyond the 16K training context.
Critical weaknesses
  1. Although the paper discusses computational complexity, empirical results on inference latency, throughput, and GPU memory usage are missing. As the number of archived states increases with context length, the additional computational and storage overhead deserves further investigation to better assess the practical efficiency of Tayra.
  2. The current ablation studies focus mainly on historical-read placement, query parameterization, and retrieval budgets. However, the individual contributions of key-derived indexing, entropy correction, and joint readout remain unclear. More targeted comparisons would help distinguish the effects of these components and better explain the performance gains over MARCH.
Questions for authors
  1. Could the authors provide empirical comparisons of inference latency, throughput, and GPU memory usage with the main baselines across different context lengths?
  2. How much do key-derived indexing, entropy correction, and joint readout individually contribute to the reported improvements? Additional ablation experiments would help clarify their respective roles.
  3. Beyond the Top-4 retrieval experiment, how does Tayra perform under different archival intervals and larger numbers of historical states? Does the overlap between cumulative snapshots affect retrieval accuracy?
1 comment

W1/Q1 — Inference efficiency and storage

We agree that practical efficiency requires empirical evaluation alongside complexity analysis. We benchmarked Transformer, Raven, Raven-based MARCH, dense Tayra, and Top-4 Tayra on a single GPU, using batch size 4, BF16 operators, and 1,024 generated tokens. Timing includes archive construction and updates, routing-key computation, index scanning, and historical reads. Tayra uses C=512. Accuracy and timing use the corresponding checkpoint and routing configuration. Mean NIAH averages six tasks at 4K, 8K, 16K.

Each context-length cell reports first-token latency (ms)/decode throughput (tokens/s)/analytical cache (GiB).

ModelMean NIAH16K32K64K
Transformer65.2558.7/608.2/8.8591727.2/426.0/17.7196115.1/263.9/35.438
Raven41.5510.7/1586.3/0.138994.8/1585.4/0.1382047.3/1585.1/0.138
Raven-based MARCH52.4964.4/746.2/4.3572250.1/634.2/8.5776307.7/466.6/17.015
Dense Tayra59.6846.6/784.8/4.5742108.4/655.6/9.0096171.3/446.2/17.880
Top-4 Tayra51.3997.3/1128.2/4.5741921.4/1122.4/9.0094105.8/1159.5/17.880

Dense Tayra gains 7.2 NIAH points over Raven-based MARCH with approximately 5% more estimated cache. First-token latency decreases by 12.2% at 16K and 6.3% at 32K. Top-4 achieves 1.44x, 1.71x, and 2.60x dense Tayra's decode throughput at 16K, 32K, and 64K, with an 8.3-point accuracy reduction. Its first-token latency is higher at 16K and lower at 32K/64K. Both variants retain the same historical archive and scan its indices. Top-4 reduces historical reading work while preserving the cache requirement. The archive continues to grow with context length.

W2/Q2 — Component contributions We added targeted comparisons to isolate key-derived indexing, entropy correction, and joint readout, together with the log-count correction. Each variant changes only the named component. Scores average six NIAH tasks at 4K, 8K, 16K, and 32K. Detailed results are provided in the appendix.

ConfigurationMean NIAH, 4K–32K
Full Tayra58.6
Replace indexing with anchor keys51.8
Remove entropy correction38.4
Replace joint readout with additive fusion38.9
Remove log-count correction33.7

Replacing key-derived indexing reduces mean accuracy by 6.9 points, supporting indices constructed from stored snapshot keys. This variant retains entropy correction and joint normalization. Removing entropy correction reduces accuracy by 20.3 points, supporting its role in routing-score calibration. MARCH-style additive fusion yields a 19.7-point reduction, supporting shared normalization of current and historical readouts. Removing log-count correction yields a 25.0-point reduction. The correction adjusts historical scores for the selected snapshot count, preventing candidate multiplicity alone from biasing joint readout toward history. Differences use unrounded scores. These controls support the components' contributions within this configuration.

Q3 — Archival intervals, overlap, and reading budgets

Archival intervals. We trained separate models with intervals of 128, 256, 512, and 1024 tokens, each using 256 slots and five middle read layers. Scores average six NIAH tasks equally at 4K, 8K, 16K, and 32K.

IntervalMean NIAH, 4K–32K
12845.4
25658.6
51255.5
102456.1

At 32K, these intervals yield approximately 32–256 snapshots per read layer. Performance is non-monotonic, with C=256 achieving the highest mean. Thus, storing more snapshots does not automatically improve retrieval. One possible explanation is that overly frequent snapshots, such as C=128, capture similar versions of cumulative memory and add redundancy with limited retrieval value. This suggests a trade-off between temporal resolution and snapshot redundancy.

Snapshot construction. A dedicated ablation uses C=512 and all 21 read layers. Chunk-local constructs snapshots from each chunk's writes while preserving continuous online recurrence. Chunk-reset additionally resets the online state.

ConstructionMean NIAH, 4K–32K
Cumulative51.8
Chunk-local39.8
Chunk-reset34.4

The cumulative-versus-Chunk-local comparison supports cumulative representations with continuous online recurrence preserved. Chunk-reset additionally tests online-state resetting. These results support cumulative snapshots despite their shared history.

Reading budgets. We varied the budget within the same dense C=256 checkpoint without retraining. Scores average six NIAH tasks at 16K and 32K.

BudgetMean NIAH, 16K/32K
Top-423.9
Top-1636.2
All snapshots48.6

Reading additional cumulative snapshots improves retrieval performance under this fixed checkpoint. This experiment measures the dense-trained model's sensitivity to inference-time truncation.

Reviewer_YjSCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Tayra gives Raven access to earlier versions of its recurrent memory. Every 512 text tokens, it saves the cumulative state and builds a compact index from its keys. A routing query scores the snapshots, then Raven’s usual query reads their contents. The main model combines the current output with reads from all available snapshots. A separately trained Top-4 version reads at most four per head and query.

Critical strengths
  1. Tayra adapts the centroid-and-entropy index from HiLS to the key slots in a recurrent state. A new query can score the snapshot using that index alone. Once the snapshot is selected, the reader still has access to the original keys and values. This is a good use of the structure Raven already gives its memory.
  2. Both models in the layer-placement experiment archive states at every layer and consult history at five layers. Five consecutive middle layers give better average recall on the needle tasks than five spaced-out layers.
Critical weaknesses
  1. The dense versus Top-4 comparison does not isolate the quality of snapshot ranking. Even with exact state-level scores, a four-snapshot budget may omit relevant information or select redundant cumulative snapshots. Moreover, the dense and Top-4 variants are trained separately, so the performance gap may also reflect differences in learned representations rather than retrieval-time selection alone. The current results do not distinguish these factors.
  2. The comparison with MARCH confounds historical indexing with the readout fusion mechanism. The Raven-based MARCH baseline adds a historical readout to the online output, with a learned null option that can suppress historical contributions. Tayra instead jointly normalizes the online and historical readouts through a shared softmax. Both allow query-dependent control of historical information, but their fusion rules differ substantially. It is therefore unclear how much of Tayra's improvement comes from the proposed indexing mechanism rather than the joint normalization.
Questions for authors
  1. How many of the four snapshots chosen by the index also rank in the top four under exact scoring? On held-out 16K examples, compare the rankings for queries with more than four available snapshots. In the trained Top-4 model, use exact slot log-partitions only to select snapshots, then apply the usual weighting and slot reads to that set. Report the average number in common and the change in multi-query or multi-value retrieval accuracy.
  2. How sensitive is dense Tayra to the query-specific weight on current memory? Estimate the average weight for each layer, head and visible snapshot count on a calibration set. On held-out sequences, replace the query-specific weight with that average and divide the remaining mass among snapshots in their original proportions. How much does next-token loss change?
1 comment

We thank the reviewer for the specific experimental suggestions. We have added analyses of snapshot selection, indexing versus fusion, and query-specific current-memory weighting.

W1/Q1 — Snapshot-ranking quality

We agree that comparing separately trained dense and Top-4 models conflates snapshot-ranking quality with differences in learned representations. We therefore performed the requested selection-only intervention on the same trained Top-4 Tayra checkpoint, with all parameters frozen and the retrieval budget fixed at four snapshots.

We evaluated 100 held-out 16K examples each from multi-query and multi-value NIAH. Exact slot log-partitions were computed for all causally visible historical snapshots and used only for selection. The selected snapshots were then processed using the original index-based weighting, joint normalization, and slot reads.

For queries with more than four visible snapshots, we measured the overlap between index-selected and exact Top-4 sets along the original model trajectory. We first averaged within each example and then across the 200 examples.

Query positionsAverage snapshots in common, out of 4
Prefill2.384
Final prompt token2.145
Decode1.996

Replacing index-based selection with exact selection improves retrieval performance under the official NIAH scoring protocol.

16K taskOriginal selectionExact selectionGain (pp)
Multi-query213716
Multi-value2428.54.5

These results identify snapshot ranking as a source of retrieval error, with exact selection yielding gains within the same trained model and at a fixed four-snapshot budget. The extended analysis covers 4K–16K contexts for both Top-4-trained and dense-trained checkpoints. Panels (a–b) compare retrieval performance under index-based and exact Top-4 selection, with full dense retrieval included in panel (b). Panels (c–d) report selection overlap during prefill, at the final prompt token, and during decoding. Snapshot selection analysis figure

Selection overlap decreases with context length, while the gains from exact selection generally increase. For the dense-trained checkpoint, exact Top-4 selection narrows the gap to full dense retrieval in most settings, but a gap remains on all evaluated tasks at 16K. Thus, ranking quality contributes to the dense–Top-4 gap, alongside potential effects of the reading budget, snapshot redundancy, and training differences.

W2 — Separating indexing from fusion

We agree that both MARCH’s learned null route and Tayra’s joint normalization provide query-dependent control of historical information. We therefore added complementary ablations that replace the indexing scheme while retaining joint normalization, or replace fusion while retaining the proposed index.

The following scores average the six NIAH tasks equally across 4K, 8K, 16K, and 32K contexts.

ConfigurationAvg. NIAH (4K–32K) ↑
Full Tayra58.6
Replace the proposed snapshot index with anchor-key indexing51.8
Replace joint normalization with MARCH-style additive fusion38.9

With joint normalization retained, anchor-key indexing lowers the mean score from 58.6 to 51.8. Retaining the proposed index while adopting MARCH-style additive fusion lowers it to 38.9. These results support contributions from both indexing and fusion within the full configuration, with possible interactions between them. Detailed results will be provided in the revised manuscript.

Q2 — Sensitivity to query-specific current-memory weighting

We performed the requested intervention on a frozen dense Tayra checkpoint, using 64 Long-Data sequences for calibration and 128 held-out sequences from disjoint documents for evaluation, all at 16K tokens.

On the calibration set, we computed the mean current-memory weight for each combination of layer, head, and visible snapshot count. During evaluation, we replaced each query’s current-memory weight with the corresponding calibration mean and redistributed the remaining probability mass among historical snapshots in their original proportions. Positions without visible snapshots retained a current-memory weight of 1.

This intervention increased next-token NLL by 0.2401 nats/token (95% CI [0.2247, 0.2551], using 10,000 paired sequence-level bootstrap resamples), equivalent to a 27.1% increase in perplexity. NLL increased on all 128 evaluation sequences, supporting the value of query-specific current/history weighting in the trained dense model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response from the Authors

General Response from the Authors

We thank the reviewers for their constructive feedback. In response, we clarified Tayra’s contribution relative to prior work and added experiments on snapshot indexing, readout fusion, archival design, and inference efficiency.

  • Method and component contributions (YjSC W2; TiFP W2/Q2; YYRK W1/Q1). Building on Raven’s recurrent key–value memory and HiLS-style indexing, Tayra uses the key slots retained in each snapshot both to index that snapshot and to address values within it. In a shared configuration, replacing stored-key indexing with anchor-derived keys reduces mean NIAH6 by 6.85 points, and replacing joint normalization with additive fusion reduces it by 19.73 points. Removing the candidate-count correction or the entropy term reduces accuracy by 24.97 and 20.27 points, respectively.

  • Snapshot ranking at a fixed retrieval budget (YjSC W1/Q1). We compared index-based and exact Top-4 selection on frozen checkpoints while retaining the original weighting and reader. At 16K, exact selection improves multi-query and multi-value retrieval by 16.0 and 4.5 points, respectively. The ranking and overlap analysis shows that better snapshot selection closes part of the gap to dense retrieval.

  • Query-dependent weighting of current memory (YjSC Q2). Replacing each query’s current-memory weight with a calibration average increases held-out next-token NLL by 0.2401 nats/token (95% CI: 0.2247–0.2551), supporting query-dependent allocation between current and historical memory.

  • Archival intervals, snapshot overlap, and read budgets (TiFP Q3). Among the four intervals tested in the same configuration, 256 tokens yields the best mean NIAH6. In a separate comparison that preserves the online recurrence, cumulative snapshots score 51.8% versus 39.8% for chunk-local snapshots. Increasing the read budget also improves recall on a fixed dense checkpoint.

  • Read placement and inference efficiency (TiFP W1/Q1; YYRK W1/Q1 and W2/Q2). In the chunk-256 comparison, restricting historical reads to five middle layers yields similar mean NIAH6 over 4K–16K to reading in all 21 layers (64.43% vs. 64.07%), with 1.56× decode throughput and 73.9% less estimated cache at 16K. Dense Tayra and Raven-based MARCH have comparable inference costs, while Tayra improves mean NIAH6 from 52.4% to 59.6%. At 64K, dense Tayra delivers 1.69× the Transformer’s decode throughput and uses 49.5% less estimated cache, with similar first-token latency. Top-4 Tayra reaches 4.39× the Transformer’s decode throughput and reduces first-token latency by 32.9%, using the same cache as dense Tayra. Across 4K–32K, mean NIAH6 is 59.6% for dense Tayra, 51.3% for Top-4 Tayra, and 65.2% for the Transformer, quantifying the accuracy trade-off.

  • Future directions. We envision scaling Tayra to larger models for long-horizon agentic tasks. We see Tayra’s ability to preserve and selectively revisit earlier recurrent states as a starting point for exploring test-time training and continual learning, including how agents might learn what experience to consolidate into memory and when to retrieve it.

We look forward to further discussion with the reviewers and would be happy to address any additional questions or comments to further improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.