- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Titan-Llama, a method for retrofitting pretrained Transformers with external neural memory modules (NMMs). It combines local attention with recurrent memory that stores historical key-value information through test-time parameter updates. The pretrained backbone remains frozen while the added modules are post-trained. Experiments on Llama-3.2-1B-Instruct report improvements over segmented attention on long-context tasks, along with reductions in inference latency and memory usage.
Critical strengths
-
Retrofitting pretrained Transformers with external neural memory is a useful direction for long-context inference. The method preserves the original backbone weights and avoids training a new language model from scratch.
-
Titan-Llama shows clear improvements over segmented attention on several long-context tasks. The reported latency and memory reductions are also promising, and the chunk-size analysis provides useful observations about the efficiency–accuracy tradeoff.
Critical weaknesses
-
The main quality experiments replace only the first four attention layers, leaving the remaining layers with full attention. These layers can still access the complete history, making it difficult to determine how much long-range information is actually recovered by the NMM. Figure 4 evaluates the efficiency of converting more layers, but corresponding task-quality results are missing. The paper therefore does not establish the quality, efficiency tradeoff when most or all attention layers are replaced.
-
The contribution of the proposed memory mechanism is not sufficiently isolated. Appendix A describes extensive post-training, including FineWeb, LongAlign, LoongRL, and QuALITY, but it is unclear whether the segmented-attention baseline receives comparable training. More importantly, there is no direct ablation disabling test-time NMM updates or replacing the memory with a static module. Without these comparisons, it is difficult to attribute the reported improvements specifically to test-time memory learning.
Questions for authors
-
Can the authors report the accuracy of the long-context task for the 5-layer, 10-layer, and 16-layer conversion settings shown in Figure 4? How much performance is preserved when all attention layers are replaced?
-
Did the segmented attention baseline undergo the same post-training process? What would happen if NMM were kept fixed during inference and not updated during testing?
-
Were the comparisons of the inference baselines conducted under equivalent cache and CUDA Graph settings? Table 1 indicates that some implementations may not use caching, which makes the reported latency comparisons difficult to interpret.
We thank the reviewer for their detailed feedback! We agree that the original experiments did not sufficiently characterize performance under more extensive layer conversion or isolate the contribution of the NMM. We have conducted additional experiments addressing both concerns and expanded our documentation of the inference implementations.
We agree that evaluating downstream performance across conversion configurations is important for understanding the quality–efficiency trade-off. We have therefore expanded our experiments to include 4, 8, 12, and 16 converted layers, replacing the original 5/10/16 configurations to align with our main experimental setting.
As shown in Table 6, F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, while F16 achieves 26.51%/23.00%, respectively. Thus, full conversion retains some downstream performance, but incurs substantial degradation under the fixed post-training budget.
We additionally report perplexity across context lengths from 8K to 128K. When trained under matched post-training budgets, perplexity increases with the ratio of converted layers but exhibits stable perplexity scaling beyond beyond the maximum sequence length seen during training. (See Figure 5 in the updated manuscript.)
We do not see this as a fundamental limitation of our method, but rather the need for increased post-training budgets for aggressive conversion configurations. We were unable to optimally tune the budget size due to computational constraints.
Our original segmented-attention baseline did not receive equivalent post-training. To address this, we have introduced a post-trained, layer-matched segmented-attention baseline that was trained with the same data mix / token budget as our main experiment.
As shown in Figure 6, Titan-Llama F4 achieves 40.50% on QuALITY and 40.80% on LongHealth, compared with 32.36% and 28.25% for the post-trained segmented-attention baseline, corresponding to improvements of 8.14 and 12.55 percentage points, respectively.
We additionally ablate the NMM across all four conversion configurations. For F4, toggling off TTT updates reduces QuALITY accuracy from 40.50% to 35.81% and LongHealth accuracy from 40.80% to 30.50%. Similar improvements from toggling writes are observed across F8, F12, and F16.
These results provide strong evidence that the memory writes contribute meaningfully to downstream performance beyond reading from static weights.
We appreciate the reviewer raising this concern and have expanded Table 3 to explicitly document the caching and CUDA Graph strategies employed by each architecture. We therefore interpret the measurements as implementation-level latency comparisons rather than fully kernel-normalized architectural benchmarks.