acceptodds
Under review as a conference paper at ICLR 2027

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

Abstract

Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV caches. Efficient recurrent alternatives reduce the asymptotic complexity of inference, but their sequential dependence limits token-wise parallelism during training, making them harder to scale. Further, the current paradigm of extending long context tasks creates competing notions of retaining information precisely and keeping long-context inference efficient. To address this gap, we introduce Titan-Llama, a post-training method for retrofitting attention-based language models with a hierarchical memory architecture that updates parameters at test-time. Titan-Llama applies a blockwise attention mask and preserves full-fidelity attention over recent tokens while compressing historical context into recurrent, fixed-size neural memory modules (NMMs). Our formulation allows pre-trained attention layers to be replaced without training a new language model from scratch. On Llama-3.2-1B-Instruct, Titan-Llama raises accuracy over segmented attention windows from 23.87% to 40.51% on QuALITY and from 14.50% to 40.75% on LongHealth. Inference benchmarks show more than 20% faster prefill, 37% faster decode at medium context lengths, and 24% lower steady-state decode memory usage in the evaluated configurations. Together, these results demonstrate that incorporating memory architecture mechanisms that utilize test-time training effectively modulates long-context behavior of pre-trained models, offering a practical route to adapting pre-trained transformers for efficient long-context inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time training, Titan, Retrofitting, Long-Context, Hybrid

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XSHtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Titan-Llama, a method for retrofitting pretrained Transformers with external neural memory modules (NMMs). It combines local attention with recurrent memory that stores historical key-value information through test-time parameter updates. The pretrained backbone remains frozen while the added modules are post-trained. Experiments on Llama-3.2-1B-Instruct report improvements over segmented attention on long-context tasks, along with reductions in inference latency and memory usage.

Critical strengths
  1. Retrofitting pretrained Transformers with external neural memory is a useful direction for long-context inference. The method preserves the original backbone weights and avoids training a new language model from scratch.

  2. Titan-Llama shows clear improvements over segmented attention on several long-context tasks. The reported latency and memory reductions are also promising, and the chunk-size analysis provides useful observations about the efficiency–accuracy tradeoff.

Critical weaknesses
  1. The main quality experiments replace only the first four attention layers, leaving the remaining layers with full attention. These layers can still access the complete history, making it difficult to determine how much long-range information is actually recovered by the NMM. Figure 4 evaluates the efficiency of converting more layers, but corresponding task-quality results are missing. The paper therefore does not establish the quality, efficiency tradeoff when most or all attention layers are replaced.

  2. The contribution of the proposed memory mechanism is not sufficiently isolated. Appendix A describes extensive post-training, including FineWeb, LongAlign, LoongRL, and QuALITY, but it is unclear whether the segmented-attention baseline receives comparable training. More importantly, there is no direct ablation disabling test-time NMM updates or replacing the memory with a static module. Without these comparisons, it is difficult to attribute the reported improvements specifically to test-time memory learning.

Questions for authors
  1. Can the authors report the accuracy of the long-context task for the 5-layer, 10-layer, and 16-layer conversion settings shown in Figure 4? How much performance is preserved when all attention layers are replaced?

  2. Did the segmented attention baseline undergo the same post-training process? What would happen if NMM were kept fixed during inference and not updated during testing?

  3. Were the comparisons of the inference baselines conducted under equivalent cache and CUDA Graph settings? Table 1 indicates that some implementations may not use caching, which makes the reported latency comparisons difficult to interpret.

1 comment

We thank the reviewer for their detailed feedback! We agree that the original experiments did not sufficiently characterize performance under more extensive layer conversion or isolate the contribution of the NMM. We have conducted additional experiments addressing both concerns and expanded our documentation of the inference implementations.

Q1: Can the authors report the accuracy of the long-context task for the 5-layer, 10-layer, and 16-layer conversion settings shown in Figure 4? How much performance is preserved when all attention layers are replaced? Q1: Figure 4 evaluates the efficiency of converting more layers, but corresponding task-quality results are missing.

We agree that evaluating downstream performance across conversion configurations is important for understanding the quality–efficiency trade-off. We have therefore expanded our experiments to include 4, 8, 12, and 16 converted layers, replacing the original 5/10/16 configurations to align with our main experimental setting.

As shown in Table 6, F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, while F16 achieves 26.51%/23.00%, respectively. Thus, full conversion retains some downstream performance, but incurs substantial degradation under the fixed post-training budget.

We additionally report perplexity across context lengths from 8K to 128K. When trained under matched post-training budgets, perplexity increases with the ratio of converted layers but exhibits stable perplexity scaling beyond beyond the maximum sequence length seen during training. (See Figure 5 in the updated manuscript.)

We do not see this as a fundamental limitation of our method, but rather the need for increased post-training budgets for aggressive conversion configurations. We were unable to optimally tune the budget size due to computational constraints.

Q2: Did the segmented attention baseline undergo the same post-training process? What would happen if NMM were kept fixed during inference and not updated during testing? W2: it is unclear whether the segmented-attention baseline receives comparable training. W2: More importantly, there is no direct ablation disabling test-time NMM updates or replacing the memory with a static module.

Our original segmented-attention baseline did not receive equivalent post-training. To address this, we have introduced a post-trained, layer-matched segmented-attention baseline that was trained with the same data mix / token budget as our main experiment.

As shown in Figure 6, Titan-Llama F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, compared with 32.36% and 28.25% for the post-trained segmented-attention baseline, corresponding to improvements of 8.14 and 12.55 percentage points, respectively.

We additionally ablate the NMM across all four conversion configurations. For F4, toggling off TTT updates reduces QuALITY accuracy from 40.50% to 35.81% and LongHealth accuracy from 40.80% to 30.50%. Similar improvements from toggling writes are observed across F8, F12, and F16.

These results provide strong evidence that the memory writes contribute meaningfully to downstream performance beyond reading from static weights.

Q3: Were the comparisons of the inference baselines conducted under equivalent cache and CUDA Graph settings? Table 1 indicates that some implementations may not use caching, which makes the reported latency comparisons difficult to interpret.

We appreciate the reviewer raising this concern and have expanded Table 3 to explicitly document the caching and CUDA Graph strategies employed by each architecture. We therefore interpret the measurements as implementation-level latency comparisons rather than fully kernel-normalized architectural benchmarks.

ModelParametersLayersGraphing and caching strategy
Full attention1.236B16Whole-forward CUDA graph; SDPA; preallocated static KV cache.
Mamba-21.344B48Upstream CUDA graphs; cached convolution and SSM states in every block.
Gated DeltaNet1.466B24Whole-forward CUDA graph; FLA fused recurrent kernels; fixed-address recurrent and convolution state buffers.
LaCT0.798B24Eager PyTorch; cached local KV, fast weights, momentum, and pending writes; updates every 2K tokens.
TTT-E2E2.898B32JAX JIT; frozen-prefix and chunk-local suffix KV caches; adaptation every 1K tokens, followed by suffix-cache rebuild.
Titan-Llama1.253B16Only the first 4 layers use CUDA graphs; remaining layers are compiled without graphs. Cached KV and recurrent-state reuse between writes.
Reviewer_jqdgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Titan-Llama adds adaptive neural memory to a frozen Llama backbone, combining local attention with compressed historical information to support efficient long-context inference.

Critical strengths
  1. Retrofitting an existing model is practically relevant. Partial conversion is a reasonable design choice, and retaining full attention in some layers does not by itself diminish the value of the approach.
  2. The experiments demonstrate useful retrieval behavior, and the paper reports important limitations, helping readers assess its applicability.
Critical weaknesses
  1. The contribution of adaptive memory remains difficult to isolate. The main configuration converts four layers, retains full attention in the remaining layers, and receives additional post-training. Baseline masks and training budgets are not clearly matched. Successful retrieval therefore does not by itself establish how much benefit comes from memory updates, since historical information may also remain accessible through the full-attention layers. The issue is attribution, rather than the use of a partially converted architecture.
  2. The quality and efficiency measurements do not establish a joint trade-off for the same configuration. Quality experiments primarily convert four layers, whereas efficiency measurements use five, ten, or sixteen. Converting additional layers may change both retrieval behavior and computational cost, so these measurements cannot be combined into a single demonstrated operating point.
  3. The retrieval results would benefit from a clearer account of retention under continued memory updates. For a mechanism that compresses historical information, an important question is whether earlier information remains accessible as additional content is processed. It would be useful to clarify whether the existing evaluations examine retrieval distance and intervening distractor content, and what they reveal about the memory’s useful operating range.
Questions for authors
  1. Under matched attention masks and post-training budgets, how much improvement comes specifically from adaptive memory? Can appropriate controls distinguish the contribution of the memory branch from that of its online updates?
  2. Can quality, latency—including memory-update steps—and peak memory be reported for the same checkpoint and layer-conversion configuration?
  3. How does retrieval performance change as relevant information becomes more distant and additional distractor content is processed? Do the existing results reveal limitations in retaining earlier information through successive updates?
1 comment

We thank the reviewer for identifying important experimental confounds in our original submission! We have substantially expanded our evaluations to better isolate the contribution of adaptive memory and establish the relationship between model quality and inference efficiency.

Q1: Under matched masks and post-training, how much improvement comes specifically from memory updates?

Further, we perform an ablation on memory updates across all four conversion configurations. For F4, disabling TTT updates reduces QuALITY accuracy from 40.50% to 35.81% and LongHealth accuracy from 40.80% to 30.50%. Similar improvements from toggling writes are observed across F8, F12, and F16. Interestingly, even a static NMM produces improvements over fine-tuned segmented attention, which produces scores of 32.36% and 28.25%.

These results provide strong evidence that the memory writes contribute meaningfully to downstream performance beyond reading from static weights.

W2: Quality experiments primarily convert four layers, whereas efficiency measurements use five, ten, or sixteen. The joint quality–efficiency trade-off for the same configuration remains unclear. Q2: Can quality, latency—including memory-update steps—and peak memory be reported for the same checkpoint and configuration?

We agree that establishing an understanding of the accuracy-efficiency tradeoff is important, so we have changed our evaluation to configurations with 4, 8, 12, and 16 converted layers. Figure 4 now reports prefill latency, decode throughput, steady-state decode memory, and peak prefill memory for the F4/F8/F12/F16 configurations evaluated for quality in Figure 5 and Table 6. The latency measurements include NMM update operations at chunk boundaries.

We additionally report perplexity across context lengths from 8K to 128K and downstream accuracy on QuALITY and LongHealth for these configurations. When trained on a post-training budget matched basis, perplexity increases with the ratio of converted layers but exhibits stable scaling to long sequence lengths far beyond the maximum sequence length seen during training. (See Figure 5 in the updated manuscript.)

Our results demonstrate a clear trade-off. The 4-layer configuration achieves the strongest downstream performance (40.50% on QuALITY and 40.80% on LongHealth), while replacing additional layers results in higher perplexity and generally lower downstream accuracy. We do not see this as a fundamental limitation of our method, but rather the need for more experimentation with post-training budgets for aggressive conversion configurations. We were unable to optimally tune the budget size due to computational constraints.

W1: Baseline masks and training budgets are not clearly matched, limiting attribution of improvements to adaptive memory.

Though we did not specify the baseline masks in the original script, they were matched to the number of converted layers.

To sufficiently distinguish the benefits of neural memory, we have introduced a segmented-attention baseline matched on both post-training budget / data mix and attention mask.

and evaluated it against Titan-Llama on QuALITY and LongHealth. Titan-Llama F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, compared with 32.36% and 28.25%, respectively, for the segmented-attention baseline. These results strongly suggest that the NMM outperforms the post-trained segmented-attention baseline.

Reviewer_F2nnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Titan-Llama, which replaces long-range attention in some Llama layers with a small memory network that updates during inference. It improves long-context QA over local attention alone and reduces inference costs, though it still falls short of full attention on some retrieval tasks.

Critical strengths
  1. The idea of incorporating test-time neural memory into a frozen pretrained LLM is interesting and well motivated by the cost of long-context attention.

  2. The paper is well written and easy to follow, with a clear presentation of the method.

  3. The experimental results are strong, showing substantial gains over segmented attention on long-context QA, alongside reductions in inference latency and memory usage.

Critical weaknesses
  1. The experimental setup requires clarification. Accuracy and efficiency are evaluated with different numbers of converted layers, leaving the trade-off between them insufficiently established. It is also unclear whether the baselines receive comparable post-training.

  2. The retrieval results show a substantial gap to full attention on some tasks. At 32K, multi-query accuracy is only 34.5%, compared with 83.5% for full attention.

Questions for authors

Please see the weaknesses above.

1 comment

We thank the reviewer for their positive assessment of our method and experimental results! We appreciate the suggestions and have conducted additional experiments to better characterize Titan-Llama's quality–efficiency trade-off and contextualize its retrieval performance.

W1a: Accuracy and efficiency are evaluated with different numbers of converted layers, leaving the trade-off between them insufficiently established.

We agree that establishing an understanding of the accuracy-efficiency tradeoff is important, so we have changed our evaluation to configurations with 4, 8, 12, and 16 converted layers. We additionally report perplexity across context lengths from 8K to 128K and downstream accuracy on QuALITY and LongHealth.

Our results demonstrate a clear trade-off. The 4-layer configuration achieves the strongest downstream performance (40.50% on QuALITY and 40.80% on LongHealth). When trained on a post-training budget matched basis, perplexity increases with the ratio of converted layers but exhibits stable scaling to long sequence lengths far beyond the maximum sequence length seen during training. (See Figure 5 in the updated manuscript.)

We do not see this as a fundamental limitation of our method, but rather the need for more experimentation with post-training budgets for aggressive conversion configurations. We were unable to optimally tune the budget size due to computational constraints.

W1b: It is also unclear whether the baselines receive comparable post-training.

In the original comparisons, baselines did not receive post-training. To address this, we have introduced a segmented-attention baseline matched on both post-training budget / data mix and attention mask.

Titan-Llama F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, compared with 32.36% and 28.25%, respectively, for the mask matched segmented-attention baseline. This provides strong evidence that segmented attention, even with matched post-training, cannot overcome the lack of global dependencies.

W2: The retrieval results show a substantial gap to full attention on some tasks. At 32K, multi-query accuracy is only 34.5%, compared with 83.5% for full attention.

We acknowledge that multi-query retrieval remains challenging relative to full attention, particularly at 32k context. To better contextualize this limitation, we have expanded our RULER comparisons to include Mamba-2, Gated DeltaNet, LaCT, and TTT-E2E across the single-needle retrieval categories.

Importantly, Titan-Llama achieves 34.5% multi-query accuracy at 32K, compared with 0.0%, 6.5%, 8.0%, and 15.3% for these respective efficient architectures. On multi-value retrieval at 32K, Titan-Llama achieves 53.0%, compared with 0.0%, 5.8%, 6.0%, and 17.5%. These results demonstrate that, although a substantial gap to full attention remains on multi-query retrieval, Titan-Llama preserves considerably more retrieval capability than the evaluated efficient alternatives.

ModelSingle needle 4K8K16K32KMulti-key 4K8K16K32KMulti-query 4K8K16K32KMulti-value 4K8K16K32K
Mamba-233.333.30.00.00.70.00.00.00.00.00.00.00.00.00.00.0
Gated DeltaNet80.748.727.711.08.77.73.72.041.018.26.86.541.216.89.05.8
LaCT87.068.746.735.029.316.08.73.339.824.214.58.038.823.514.06.0
TTT-E2E90.091.346.324.376.056.331.717.380.870.023.815.369.855.037.017.5
Titan-Llama97.396.088.078.791.388.066.739.396.590.570.534.588.081.063.053.0

Comments on the paper

1 comment

We thank all three reviewers for their thoughtful feedback and constructive suggestions! In response, we have substantially expanded our experimental evaluation, conducting over a dozen additional experimental comparisons and ablations and revising the manuscript to address the concerns raised. We summarize the main changes below.

1. Characterizing architectural scaling under a fixed post-training budget

Concern: All three reviewers noted that our original experiments evaluated quality and efficiency using different numbers of converted layers, making it difficult to understand how performance changes as more attention layers are replaced.

Response: To address this, we have expanded our experiments to evaluate F4, F8, F12, and F16 configurations, reporting downstream accuracy on QuALITY and LongHealth, perplexity across 8K–128K contexts, and corresponding inference efficiency measurements (Figures 4–5 and Table 6). Importantly, we hold the post-training budget fixed across all four configurations to provide a controlled comparison of how performance scales with the number of converted layers.

We would like to distinguish this characterization from the practical quality–efficiency frontier of our method. In a real-world deployment, configurations with more converted layers would likely receive greater post-training investment to recover the capabilities of the pretrained backbone. Establishing that frontier would require independently optimizing training budgets and hyperparameters for each configuration, which is beyond our available computational resources.

Under our fixed-budget setting, we observe that more aggressive conversion incurs a trade-off in absolute performance, while several configurations exhibit favorable perplexity trends as context length increases well beyond the training horizon. These results highlight the ability of Titan-Llama to maintain stable length-wise generalization even as a greater proportion of attention layers is replaced. We believe this controlled evaluation provides valuable insight into the scalability of our architecture and establishes a foundation for further improvements through configuration-specific post-training.

2. Isolating the contribution of neural memory and test-time updates

Concern: Reviewers questioned whether the observed gains could be attributed to additional post-training or retained full-attention layers rather than the proposed NMM mechanism. They requested comparisons with matched attention masks and training budgets, as well as an ablation of test-time memory updates.

Response: We introduced a layer-matched, token-budget-matched segmented-attention baseline, post-trained with LoRA. Titan-Llama F4 outperforms this baseline by 8.14 percentage points on QuALITY and 12.55 points on LongHealth (Figure 6).

We additionally ablated test-time NMM updates across all four conversion configurations (Table 6). For F4, disabling updates while retaining the memory module reduces QuALITY accuracy from 40.50% to 35.81% and LongHealth accuracy from 40.80% to 30.50%. These results provide direct evidence that test-time memory updates contribute meaningfully to performance beyond static memory retrieval.

3. Strengthening baseline comparisons and inference methodology

Concern: Reviewers highlighted the gap to full attention on difficult RULER retrieval tasks and questioned whether latency comparisons were fair given differences in caching and CUDA Graph implementations.

Response: We expanded our RULER evaluation to include Mamba-2, Gated DeltaNet, LaCT, and TTT-E2E across single-needle, multi-key, multi-query, and multi-value retrieval. Although Titan-Llama continues to exhibit a gap relative to full attention on multi-query retrieval at 32K, it achieves 34.5% accuracy compared with 0.0%, 6.5%, 8.0%, and 15.3% for the four efficient alternatives, respectively. These experiments provide important context for the strengths and limitations of our architecture.

We have also expanded Table 3 to document each model's caching strategy, CUDA Graph coverage, and inference implementation, and updated our latency comparisons accordingly. While implementations differ in their degree of optimization, we clarify these differences to make the comparisons more transparent and reproducible.

Overall, we believe these additional experiments comprehensively address the reviewers' concerns by isolating and ablating the contributions of neural memory and test-time training, establishing a controlled quality-efficiency trade-off across converted layers under a fixed post-training budget, and demonstrating Titan-Llama's advantages over strengthened baselines through expanded evaluations and providing more details on inference benchmarking. Together, these results substantially strengthen the experimental evidence for our method and clarify both its practical benefits and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.