acceptodds
Under review as a conference paper at ICLR 2027

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

Abstract

Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires fewer retrieval queries and fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Parametric Memory, Chunk-Level Supervision, Memory Fusion, Knowledge Injection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XtT4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Deep Fusion Memory (DFM), a chunk-based parametric memory method for large language models (LLMs). DFM is designed to reduce the supervised construction costs associated with traditional parametric memory and to improve how stored information is integrated. The approach utilizes chunk embeddings—obtained via offline retrieval—to supervise a "Memory Predictor." This predictor then injects the retrieved memory into the intermediate layers of a frozen LLM through a gated cross-attention mechanism. The model is trained by jointly optimizing both the language modeling loss and the memory prediction loss, eliminating the need for online retrieval during inference. Experiments conducted across various knowledge-based question answering and domain adaptation tasks demonstrate that DFM enhances overall performance while simultaneously lowering the overhead of offline retrieval and reducing training costs.

Critical strengths
  1. The research problem holds practical significance. This paper addresses the challenge of knowledge enhancement in large language models (LLMs). It proposes improvements to existing parametric memory methods—specifically by mitigating the high cost associated with constructing supervised data—thereby offering a clear research motivation and tangible practical value.
  2. The overall methodology is well-designed and demonstrates a degree of innovation. By integrating "chunk-level memory prediction" with gated cross-attention, the authors train additional memory modules while keeping the LLM backbone parameters frozen. The framework is comprehensive and eliminates the need for online retrieval during the inference phase.
Critical weaknesses
  1. There is a lack of ablation studies regarding the supervision of memory prediction. The added memory module contains approximately 750 million parameters; therefore, it is unclear whether the observed performance gains stem from the memory module itself or simply from the increased model capacity and additional language modeling training. Regarding the MSE metric specifically, while the coordinate-wise MSE of the normalized targets might be small, one cannot conclude that this branch is ineffective based solely on the loss value. Further ablation studies or empirical evidence of optimization are required.
  2. Section 3 mentions that the memory for the current chunk is predicted using the preceding chunk during training. However, Appendix A.1 indicates that during QA inference, the model utilizes the currently visible prompt chunk, and the memory is not updated continuously during the generation process. A more detailed explanation regarding the correspondence between memory and token positions is necessary.
  3. A significant concern is that, while the authors explicitly state in the title that their proposed approach is intended for large language models (LLMs), their experiments are limited to only two models—Qwen3 and Mistral—in their "base" versions. The absence of comparisons with other models, such as DeepSeek or GLM, leads me to question the generalizability of the proposed method.
Questions for authors
  1. The actual contribution of the supervision applied to retrieved memory needs to be verified. It is recommended that the authors conduct control experiments—such as setting β=0\beta=0 or using randomly shuffled retrieval targets—while keeping the model architecture and training budget constant. This would help isolate the specific contribution of retrieval supervision to performance improvements. Furthermore, an analysis of the alignment between the predicted memory and the actual retrieval targets is warranted to confirm whether the model has genuinely learned to retrieve relevant knowledge.
  2. The independent contribution of "Deep Fusion" requires verification. While Figure 3 examines the impact of injecting cross-attention at different layers, it does not provide a direct comparison between "deep fusion" and fusion at the output layer. Given that the Deep Fusion Mechanism (DFM) and the MLP-based memory differ in terms of their supervision objectives, model architecture, and parameter scale, it is difficult to attribute the current results solely to "Deep Fusion." It is suggested that the authors compare cross-attention at intermediate layers against fusion at the output layer—while maintaining consistent memory prediction mechanisms and comparable parameter budgets—to validate the specific advantages of the Deep Fusion approach.
  3. Why did the authors choose only these two large language models (Qwen3 and Mistral) ?
Reviewer_khapOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Deep Fusion Memory (DFM), a chunk-level parametric memory architecture that enables large language models to acquire and utilize external knowledge without online retrieval. Specifically, the authors first partition training sequences into non-overlapping chunks and construct chunk-level memory targets by retrieving the top-K relevant chunk embeddings from an external corpus using the preceding chunk as the query. Next, a causal memory predictor is trained to generate continuous memory representations from the preceding context, supervised by the retrieved memory embeddings. The predicted memory is then projected into the language model's hidden space and integrated into selected decoder layers through gated cross-attention, allowing tokens within the same chunk to share memory while adaptively controlling its contribution. The memory predictor and fusion modules are jointly optimized using language modeling and memory prediction losses, while keeping the backbone frozen. Experiments on knowledge-intensive question answering and domain adaptation tasks demonstrate performance improvements over competing methods, along with reduced training costs and substantially fewer retrieval operations for constructing memory supervision. However, I have some concerns about this paper. My detailed comments are as follows.

Critical strengths
  1. The authors propose a chunk-level parametric memory architecture that jointly trains a causal memory predictor and memory fusion modules while keeping the LLM backbone frozen, enabling the model to utilize learned external knowledge without online retrieval.
  2. The proposed method incorporates predicted memory into intermediate decoder layers through gated cross-attention, allowing tokens within the same chunk to share memory representations while adaptively controlling their contributions to token prediction.
  3. The experiments are conducted on multiple LLM backbones across six knowledge-intensive QA benchmarks and three domain adaptation areas, demonstrating consistent improvements in average performance over the compared baselines.
Critical weaknesses
  1. Insufficient evaluation of capability preservation. Although DFM freezes the LLM backbone to mitigate catastrophic forgetting, the added memory modules can still affect the model's original capabilities. More comprehensive evaluations on general-purpose and previously learned tasks are needed to support the capability preservation motivation.
  2. Lack of end-to-end inference efficiency evaluation. Avoiding online retrieval is a key motivation, yet the paper does not compare DFM's inference efficiency with RAG or RETRO. The additional memory predictor and cross-attention modules introduce computational overhead, making latency, throughput, and memory usage comparisons necessary.
  3. Insufficient justification of the parametric memory mechanism. DFM resembles a retrieval-regularized, context-conditioned adapter rather than a clearly distinguishable parametric memory mechanism. The paper does not demonstrate whether the predicted representations actually encode external factual knowledge or merely facilitate representation adaptation. In particular, the absence of a β = 0 ablation makes it difficult to isolate the contribution of retrieval supervision from that of the additional trainable modules.
  4. Potential information loss from mean pooling. DFM constructs chunk representations by mean-pooling token embeddings, which discards token order and may lose important semantic information, particularly entity relationships. The paper lacks comparisons with more expressive alternatives, such as attention pooling or lightweight encoders.
  5. Inconsistency between training and inference. During training, memory queries are constructed from preceding chunks, whereas QA inference uses the current visible prompt chunks (Appendix A.1). Moreover, memory is not updated during answer generation. This discrepancy may introduce a train–inference mismatch, but its impact is not investigated.
  6. Insufficient justification of the memory predictor architecture. DFM introduces approximately 0.75B additional parameters, yet Appendix C.4 shows that a 14-layer predictor slightly outperforms the default 28-layer predictor (27.31 vs. 27.21). The paper lacks a thorough analysis of the trade-off between predictor complexity, computational cost, and performance.
Questions for authors

See Weaknesses.

1 comment

Thank you for the detailed response and additional experiments, which largely address my concerns regarding capability preservation, inference efficiency, and mean pooling. However, the β = 0 results on Mistral-7B still raise questions about the distinction between parametric memory and retrieval-regularized model adaptation. Moreover, the impact of the training–inference discrepancy remains experimentally unverified, and the choice of the more expensive 28-layer predictor is insufficiently justified. Overall, while the additional experiments strengthen the empirical evaluation, my main methodological concerns remain partially unresolved. Therefore, I maintain my original rating.

Reviewer_LpvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Deep Fusion Memory (DFM), a parametric memory that adds external knowledge to a frozen LLM without retrieval at inference. Training sequences are split into fixed-size chunks. For each chunk, the nearest neighbors of the preceding chunk are retrieved offline and stored as embeddings (target memory). A causal memory predictor learns to predict these embeddings from the preceding text. The predicted memory is projected into the hidden space and fused into a few intermediate and late decoder layers through gated cross-attention. The predictor and fusion modules are trained end-to-end with language modeling and memory-prediction losses while the backbone stays frozen. Compared with token-level parametric memory such as MLP Memory, DFM needs far fewer retrieval queries and retrieved items to build its training data, and it uses fewer added parameters and less training time. On single-hop and multi-hop QA benchmarks, DFM achieves the best average over non-parametric and parametric baselines on Mistral and Qwen3 backbones. It also improves the average performance on code, math and law adaptation.

Critical strengths
  • The problem is relevant and the design is simple. Chunk-level supervision is kind of a practical way to cut the large target-construction cost of token-level parametric memory. Fusing memory into hidden states is a natural improvement over output-probability interpolation.
  • The results are consistent and the analysis is broad. DFM has the best average across three backbones and across QA and domain adaptation. The paper includes many useful ablations (chunk size, fusion layers, memory slots, gating, predictor depth, loss, projector sharing), clear efficiency accounting, and helpful case studies.
Critical weaknesses
  • It is not clear what drives the gains, and the knowledge-injection claim is not well supported. The training corpus is Wikipedia 2018, which is almost certainly in the pretraining data of all backbones, so the paper does not show that DFM adds new knowledge rather than reactivating known facts. There is no ablation without memory supervision or with random targets, so the gain may come from adding 0.75B trainable parameters trained on Wikipedia. How well the predictor actually reproduces the target memory is never measured. The targets may also contain the query chunk itself or text from the same article, since the datastore holds training chunks and the top-16 keeps the nearest neighbor. At inference, memory is queried from the current prompt chunk instead of the preceding chunk used in training. The main motivation, avoiding inference-time retrieval cost and forgetting, is not measured.
  • The baselines and evaluation are weak. RAG uses only the top-1 passage and falls below the base model on Qwen3. RETRO is about equal to the base model on Mistral and Qwen3 but strong on OLMo-3, which suggests there might be an inconsistent implementation. CPT and LoRA fall well below the base model on several tasks, which suggests a format or tuning problem (?). The closest prior work (Memory Decoder, Memory Decoder at Scale, MemSFT) is cited but not compared, and related in-model memory methods such as Memory Layers at Scale and KBLaM are missing. All results are single runs, the absolute gains are about 3 points, and many differences in the average and ablations are driven by Bamboogle, which has only 125 questions.
Questions for authors
  • Can you evaluate DFM on knowledge that the backbone has not seen, for example post-cutoff Wikipedia, synthetic facts, or a private corpus? This would directly test the knowledge-injection claim.
  • What happens with β = 0, with random or shuffled target memory, or with a parameter-matched module trained only with the LM loss? How close are the predicted memories to the targets on held-out text?
  • Are the query chunk itself and chunks from the same article excluded from the training targets?
  • RAG uses only the top-1 retrieved passage. What is the size of the retrieved unit, and how does RAG perform with top-k passages (e.g., k = 5)?
  • How was RETRO applied to Mistral and Qwen3, and was it trained? It scores close to the base model there but much higher on OLMo-3 (App. B.1). CPT and LoRA fall below the base model on several tasks; could this come from the QA prompt format or the training setup? How does DFM compare with Memory Decoder or memory-layer methods (if possible)?
  • What are the inference latency, throughput and memory overhead of DFM compared with the base model and RAG? What is the wall-clock cost of target construction?
  • From an associative-memory view, the predictor stores the corpus in parameters, and the cross-attention over K slots is a one-step content-addressable readout, similar to a modern Hopfield update. How does performance change as the corpus grows (capacity and interference)? Would iterative or energy-based retrieval over the memory slots help multi-hop reasoning?
  • Does adding memory hurt general abilities, for example on MMLU or other standard benchmarks? Why do some LawBench tasks drop sharply relative to the base model?
  • Please report multiple seeds or confidence intervals. Were the fusion layers [9, 16, 24, 31] tuned for the 36-layer Qwen3 models or reused from the 32-layer Mistral? Why keep gate bias −2 and a 28-layer predictor when bias 0 and 14 layers work better?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.