- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LAMP, a dual-pathway latent memory injection framework for frozen language models. A query-conditioned encoder compresses external evidence into latent tokens, which are projected into the KV cache of each transformer layer. In parallel, cross-attention adapters inject token-level evidence features into the residual stream of selected upper layers. The framework trains the memory encoder, projector, and adapters jointly using answer supervision, context distillation, and multiple auxiliary objectives, while keeping the backbone language model frozen.
The authors evaluate LAMP on eight question-answering and knowledge-editing benchmarks using two frozen language models. The results show improvements over conventional in-context injection and a modest F1 advantage over a LoRA-fine-tuned in-context baseline. Additional experiments investigate the contributions of the two injection pathways, their roles during reading and generation, robustness to distractors, retrieval from large corpora, and mid-generation evidence updates. Overall, the paper presents an interesting approach to external memory injection and provides useful empirical observations about the complementary roles of KV-based and residual-based information delivery.
Critical strengths
Interesting architectural design. The paper introduces two complementary injection pathways that deliver external evidence through the KV cache and upper-layer residual stream. The proposed architecture provides an interesting perspective on how compressed evidence can be incorporated into frozen language models.
Comprehensive evaluation across different scenarios. The experiments cover multiple question-answering and knowledge-editing benchmarks, together with investigations of distractor robustness, long-corpus retrieval, inference efficiency, and mid-generation evidence updates. The paper also reports useful negative results and limitations, including the degradation of single-memory compression under distractors.
Critical weaknesses
Overly complicated training objectives and insufficient ablation studies. The training objective in Eq. (1) combines ten loss terms with numerous additional hyperparameters, making the training procedure unnecessarily complex. Although Appendix E defines these objectives, the necessity and individual contributions of several regularization and auxiliary losses remain insufficiently justified. Moreover, the paper lacks systematic loss-level ablations, making it difficult to determine which training components are essential to the reported improvements.
Inconsistent accuracy improvements over the in-context baseline. Tables 1 and 14 show that LAMP achieves lower accuracy than the in-context baseline in several settings, despite its improvements in F1. For memory-based tasks, effectively utilizing retrieved evidence to generate accurate answers is essential. Therefore, higher F1 scores alone do not necessarily demonstrate superior memory utilization, particularly when answer accuracy remains lower than that of the baseline.
Insufficient evidence for the proposed functional separation between the two pathways. The evidence crossover experiment primarily examines the KV-only configuration, where answer recall is substantially reduced after disabling the residual pathway. However, this does not establish whether the KV memory preserves answer-specific information or whether the model simply fails to extract it without the residual pathway. Thus, the proposed functional separation between the two pathways remains insufficiently validated.
Questions for authors
-
Interpretation of CE in the No-harm Loss: Could the authors clarify the intuitive meaning of cross-entropy (CE) in the no-harm loss and explain how this objective prevents memory injection from degrading the model's original performance?
-
Ablation of Regularization Losses: Could the authors provide experimental results showing how model performance changes when the regularization losses are removed? In particular, how much do these losses contribute to the overall performance of LAMP?
-
Further Validation of Pathway-Specific Roles: Could the authors provide a more systematic evidence crossover analysis comparing KV-only, residual-only, and full LAMP under both own-evidence and donor-evidence conditions? Combining these comparisons with phase-specific interventions would help clarify the respective contributions of the two pathways to evidence reading, memory representation, and answer extraction.
Halfway summary of the discussion
Update, 18 November. This summary was written earlier in the discussion, before any reviewer had replied. All three reviewers have since responded, and Reviewer qMae has raised the rating. Please see our closing summary, posted today, for where the submission now stands. The text below is kept as originally posted.
We thank the AC and the reviewers. We posted our responses and two new experiments about a week ago, and since we are more than halfway through the rebuttal period, we would like to summarise here where the discussion stands. We have also uploaded our revised paper based on the reviews, and all changes are highlighted in blue.
1. What all three reviewers agree on
No review questions the novelty of the design, names prior work that already does this, or disputes a number in the paper. We take this agreement as clear support for the value of the work.
2. The concerns are about interpretation and reporting, and all are resolved
None of them affect the results and strengths above. For each, we note what the paper already contained and what we added.
3. Summary
We do not think the current ratings follow from what the reviews themselves say, and none has been revisited since our responses. We respectfully ask the AC to weigh the reviews by their content and by this discussion.