- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how Memory Decoder scales and evaluates the ability of pretrained memory to capture general and domain-specific knowledge. By introducing a distributed FAISS pipeline and sparse kNN distribution storage, the work addresses retrieval scalability and storage overhead. Experiments show that, with general memory, Pythia-410M paired with a 6.9B-parameter memory surpasses Pythia-12B. For domain-specific memory, Memory Decoder outperforms CPT, LoRA, and RAG.
Critical strengths
-
This work effectively scales Memory Decoder along both the model and data dimensions. The authors scale the external memory to 6.9B parameters and pretrain it on 300B tokens. To support this scale, they develop a distributed FAISS pipeline that reduces the cost of large-scale retrieval through embedding compression, index sharding, and parallel search, while sparse kNN distribution storage retains only the nonzero support.
-
General memory delivers consistent gains across different model and memory scales. Across 17 benchmarks, memory improves performance in 47 of 51 task-scale combinations. In particular, Pythia-410M paired with a 6.9B-parameter memory achieves an average score of 37.34, surpassing Pythia-12B at 37.24 while using 39% fewer total parameters.
Critical weaknesses
-
The comparison between Memory Decoder and methods such as CPT and LoRA is not entirely fair. Although the authors control for training cost, Memory Decoder requires substantially more computation at inference time than CPT and LoRA. The paper does not compare Memory Decoder with these methods under matched inference-time compute. In addition, the authors state that "LoRA is applied to the query, key, value, and MLP layers, with its rank adjusted to match the parameter count of our memory modules." However, the experiments include a setting with a 0.6B model and a 1.7B memory, whereas the number of parameters fine-tuned by LoRA cannot exceed 0.6B in this case. The authors should clarify how a fair parameter-matched comparison was implemented in this setting.
-
The components added to the Memory Decoder framework, such as FAISS and sparse kNN distribution storage, are relatively mature and commonly used techniques, which limits the methodological novelty. Moreover, the scaling experiments offer limited broader insight beyond the empirical observation.
Questions for authors
Could the authors provide a comparison between Memory Decoder and methods such as CPT and LoRA under the same inference-time compute budget? It would also be helpful if the authors could provide more details on the choice of LoRA rank in the 0.6B model + 1.7B memory setting, particularly how parameter matching is handled in this case.