acceptodds
Under review as a conference paper at ICLR 2027

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Abstract

Decoder-only language models (LMs) entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces an external parametric long-term memory and shows primitive results, albeit on small scales. In this work, we present Memory Decoder at Scale and systematically study memory scaling with model and data size, parameter allocation between memory and the LM, and pretrained memory's ability to capture general and domain-specific knowledge. We scale external memory to 6.9B parameters and pretrain it on 300B tokens. At this scale, we address retrieval scalability and storage overhead through a distributed pipeline for FAISS indexing and retrieval and sparse kNN distribution storage, respectively. Our results show that pairing larger memories with smaller LMs outperforms scaling the LM alone. This architectural disentanglement of long-term memory from the LM substantially improves parameter efficiency. A 6.9B general memory enables Pythia-410M to surpass Pythia-12B on average across 17 benchmarks with 39% fewer total parameters. Domain-specific memory outperforms the strongest reported baseline among CPT, LoRA, and RAG by 4.05–8.53 points on average. Overall, our results establish the architectural advantage of external memory over conventional decoder-only LMs, paving the way toward disentangling long-term memory from the LM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parametric memory, long-term memory, pretrain

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CDzCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how Memory Decoder scales and evaluates the ability of pretrained memory to capture general and domain-specific knowledge. By introducing a distributed FAISS pipeline and sparse kNN distribution storage, the work addresses retrieval scalability and storage overhead. Experiments show that, with general memory, Pythia-410M paired with a 6.9B-parameter memory surpasses Pythia-12B. For domain-specific memory, Memory Decoder outperforms CPT, LoRA, and RAG.

Critical strengths
  1. This work effectively scales Memory Decoder along both the model and data dimensions. The authors scale the external memory to 6.9B parameters and pretrain it on 300B tokens. To support this scale, they develop a distributed FAISS pipeline that reduces the cost of large-scale retrieval through embedding compression, index sharding, and parallel search, while sparse kNN distribution storage retains only the nonzero support.

  2. General memory delivers consistent gains across different model and memory scales. Across 17 benchmarks, memory improves performance in 47 of 51 task-scale combinations. In particular, Pythia-410M paired with a 6.9B-parameter memory achieves an average score of 37.34, surpassing Pythia-12B at 37.24 while using 39% fewer total parameters.

Critical weaknesses
  1. The comparison between Memory Decoder and methods such as CPT and LoRA is not entirely fair. Although the authors control for training cost, Memory Decoder requires substantially more computation at inference time than CPT and LoRA. The paper does not compare Memory Decoder with these methods under matched inference-time compute. In addition, the authors state that "LoRA is applied to the query, key, value, and MLP layers, with its rank adjusted to match the parameter count of our memory modules." However, the experiments include a setting with a 0.6B model and a 1.7B memory, whereas the number of parameters fine-tuned by LoRA cannot exceed 0.6B in this case. The authors should clarify how a fair parameter-matched comparison was implemented in this setting.

  2. The components added to the Memory Decoder framework, such as FAISS and sparse kNN distribution storage, are relatively mature and commonly used techniques, which limits the methodological novelty. Moreover, the scaling experiments offer limited broader insight beyond the empirical observation.

Questions for authors

Could the authors provide a comparison between Memory Decoder and methods such as CPT and LoRA under the same inference-time compute budget? It would also be helpful if the authors could provide more details on the choice of LoRA rank in the 0.6B model + 1.7B memory setting, particularly how parameter matching is handled in this case.

Reviewer_Ah48OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper scales Memory Decoder, a standalone parametric memory trained to mimic kNN retrieval distributions, to 6.9B parameters and 300B training tokens. Scalability is enabled by a distributed FAISS pipeline and sparse storage of kNN targets.

The main result is that a small LLM paired with a large memory can outperform a larger LLM alone: Pythia-410M + Mem-6.9B slightly exceeds Pythia-12B on average across 17 benchmarks while using 39% fewer parameters. Domain-specific memories for biology, law, and finance also outperform CPT, LoRA, and RAG baselines.

Critical strengths
  • Strong systems contribution. The paper introduces a scalable retrieval pipeline with shard-aware routing and efficient sparse storage of kNN targets.

  • Comprehensive evaluation. Results cover 17 general benchmarks, three domains, multiple model families and scales.

  • Well-controlled CPT comparison. In Table 4 they match data, compute,capacity and (interpolation) interface and the proposed approach ("Memory") consistently outperforms CPT - with the largest gains on BioInst (minor).

Critical weaknesses
  • Practical costs are unclear. The paper does not report inference latency, throughput, or the aggregate offline storage footprint of the retrieval index and kNN training targets.

  • The headline advantage could be perceived as small. The key result exceeds Pythia-12B by only 0.10 points, and no variance estimates are provided for this comparison - although the total parameter reduction by 39% is very important. (minor)

Questions for authors
  • Can you provide more details on the practical costs of implementing and utilizing the proposed memory architecture?

  • Would it be possible to extrapolate performance gains for the same total parameter count in a system consisting partly of a (small) LLM and a (larger) memory? For example drawing vertical dotted arrows in Figure 1.

Reviewer_YhZHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work extends the Memory Decoder paper, focusing on how to scale it to 300B training tokens and (parametric) memory size up to 6.9B parameters. Experiments validated the approach by pairing the memory model with the backbone model in various sizes. Memory Decoder is a “standalone parametric memory” model that distills nonparametric kNN language models using a standard, decoder-only Transformer model. It was trained separately and combined with a frozen backbone model by interpolating their next-token distributions. To handle a corpus at this scale, the authors used sparse kNN distribution storage to reduce the storage size, and designed a distributed pipeline for indexing and retrieval by leveraging FAISS. When pairing a backbone model with general memory (i.e., trained using 300B tokens), the proposed approach can outperform larger backbone-only models. When solving domain-specific tasks with domain-specific memory, the approach also on average outperformed baselines like continued pre-training (CPT), LoRA and top-5 RAG.

Critical strengths
  • The work provides a practical solution on how to scale the previous work, Memory Decoder, to a reasonably large scale.
  • The experiments generally show that the parametric memory can help the backbone model and outperform backbone-only models with more parameters.
  • Domain-specific memory is also shown to help improve performance on domain-specific tasks, compared to other standard baselines.
Critical weaknesses
  • It is not completely clear whether the better performance is completely due to the knowledge of memory, or from more training tokens or more total number of model parameters. (See some of my questions for detail).
  • The proposed approach does not have clear advantages over baselines in some experiments. For instance, LoRA outperforms memory on FinEval with Qwen3-1.7B and RAG is slightly better than memory on FinEval with Qwen3-8B (Table 2); memory consistently underperformed the backbone-only models at every scale on TruthfulQA.
  • No math, code or long-form generation tasks. The results on reasoning-style tasks are near random (e.g., LogiQA 21.7-23.0, GPQA 24.1-27.9) and thus could be noise. The approach may still work mostly on knowledge-intensive tasks.
Questions for authors
  • Q1: How are these 17 benchmarks chosen? Are they good representations of the common tasks?
  • Q2: What are some detailed configurations of FAISS (e.g., K, temperature, number of IVF centroids, nprobe, PQ code size and the number of shards)? Have the authors compared the retrieval recall of their pipeline with exact search?
  • Q3: Initialization: are the general memories initialized from pretrained Pythia checkpoints (App. B.1)? If so, are the compute and tokens used in prior pretraining included in the discussion in Table 1 and Figure 1? How does the memory model alone compare with its initialization model?
  • Q4: Total compute: what is the end-to-end compute including datastore construction: the Pythia-6.9B forward pass over the Pile, the domain Qwen3-4B CPT epoch, and FAISS indexing and search?
  • Q5: Some α\alpha values are interesting in App. G, Table 15, as α=0\alpha=0 means backbone model only and α=1\alpha=1 means memory model only. Can the authors discuss these special cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.