- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Deep Fusion Memory (DFM), a chunk-based parametric memory method for large language models (LLMs). DFM is designed to reduce the supervised construction costs associated with traditional parametric memory and to improve how stored information is integrated. The approach utilizes chunk embeddings—obtained via offline retrieval—to supervise a "Memory Predictor." This predictor then injects the retrieved memory into the intermediate layers of a frozen LLM through a gated cross-attention mechanism. The model is trained by jointly optimizing both the language modeling loss and the memory prediction loss, eliminating the need for online retrieval during inference. Experiments conducted across various knowledge-based question answering and domain adaptation tasks demonstrate that DFM enhances overall performance while simultaneously lowering the overhead of offline retrieval and reducing training costs.
Critical strengths
- The research problem holds practical significance. This paper addresses the challenge of knowledge enhancement in large language models (LLMs). It proposes improvements to existing parametric memory methods—specifically by mitigating the high cost associated with constructing supervised data—thereby offering a clear research motivation and tangible practical value.
- The overall methodology is well-designed and demonstrates a degree of innovation. By integrating "chunk-level memory prediction" with gated cross-attention, the authors train additional memory modules while keeping the LLM backbone parameters frozen. The framework is comprehensive and eliminates the need for online retrieval during the inference phase.
Critical weaknesses
- There is a lack of ablation studies regarding the supervision of memory prediction. The added memory module contains approximately 750 million parameters; therefore, it is unclear whether the observed performance gains stem from the memory module itself or simply from the increased model capacity and additional language modeling training. Regarding the MSE metric specifically, while the coordinate-wise MSE of the normalized targets might be small, one cannot conclude that this branch is ineffective based solely on the loss value. Further ablation studies or empirical evidence of optimization are required.
- Section 3 mentions that the memory for the current chunk is predicted using the preceding chunk during training. However, Appendix A.1 indicates that during QA inference, the model utilizes the currently visible prompt chunk, and the memory is not updated continuously during the generation process. A more detailed explanation regarding the correspondence between memory and token positions is necessary.
- A significant concern is that, while the authors explicitly state in the title that their proposed approach is intended for large language models (LLMs), their experiments are limited to only two models—Qwen3 and Mistral—in their "base" versions. The absence of comparisons with other models, such as DeepSeek or GLM, leads me to question the generalizability of the proposed method.
Questions for authors
- The actual contribution of the supervision applied to retrieved memory needs to be verified. It is recommended that the authors conduct control experiments—such as setting or using randomly shuffled retrieval targets—while keeping the model architecture and training budget constant. This would help isolate the specific contribution of retrieval supervision to performance improvements. Furthermore, an analysis of the alignment between the predicted memory and the actual retrieval targets is warranted to confirm whether the model has genuinely learned to retrieve relevant knowledge.
- The independent contribution of "Deep Fusion" requires verification. While Figure 3 examines the impact of injecting cross-attention at different layers, it does not provide a direct comparison between "deep fusion" and fusion at the output layer. Given that the Deep Fusion Mechanism (DFM) and the MLP-based memory differ in terms of their supervision objectives, model architecture, and parameter scale, it is difficult to attribute the current results solely to "Deep Fusion." It is suggested that the authors compare cross-attention at intermediate layers against fusion at the output layer—while maintaining consistent memory prediction mechanisms and comparable parameter budgets—to validate the specific advantages of the Deep Fusion approach.
- Why did the authors choose only these two large language models (Qwen3 and Mistral) ?
Thank you for the detailed response and additional experiments, which largely address my concerns regarding capability preservation, inference efficiency, and mean pooling. However, the β = 0 results on Mistral-7B still raise questions about the distinction between parametric memory and retrieval-regularized model adaptation. Moreover, the impact of the training–inference discrepancy remains experimentally unverified, and the choice of the more expensive 28-layer predictor is insufficiently justified. Overall, while the additional experiments strengthen the empirical evaluation, my main methodological concerns remain partially unresolved. Therefore, I maintain my original rating.