acceptodds
Under review as a conference paper at ICLR 2027

Dual-Pathway Latent Memory Injection for Frozen Language Models

Abstract

Frozen large language models (LLMs) are usually augmented with external evidence by prepending the supporting passage to the prompt. This in-context injection is simple and effective, but prefill cost grows with evidence length, and the evidence competes with the generated history for attention. Recent latent injection methods instead write evidence into the KV cache, which integrates it more directly and at lower cost, yet these methods have not matched the accuracy of in-context injection. We explain this gap through the mechanics of factual recall: transformer LLMs rely on both MLP-encoded factual associations and attention-based attribute extraction, whereas KV injection acts only on attention, so under compression it preserves broad semantics but loses the fine-grained details needed for verbalization. We propose LAMP (Latent Augmented Memory with Parallel Pathways), which compresses evidence into a fixed-size latent memory and injects it into a frozen LLM through two complementary pathways: a KV-cache pathway that broadcasts compressed semantics across layers, and a residual pathway in which the LLM's hidden states query token-level evidence features to recover missing details. On eight fictional, real-world, counterfactual, and long-tail question-answering benchmarks, LAMP outperforms in-context injection overall and also scores above an in-context baseline fine-tuned on the same data. Because each memory holds one passage, longer evidence is handled by composing memories, which keeps accuracy stable up to 256K tokens at constant memory per query. LAMP also supports mid-generation updates, adding new evidence without re-processing the preceding context and correcting earlier answers on the fly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Memory, Latent Memory Injection, Generation Time Memory Injection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bSmiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LAMP, a dual-pathway latent memory injection framework for frozen language models. A query-conditioned encoder compresses external evidence into latent tokens, which are projected into the KV cache of each transformer layer. In parallel, cross-attention adapters inject token-level evidence features into the residual stream of selected upper layers. The framework trains the memory encoder, projector, and adapters jointly using answer supervision, context distillation, and multiple auxiliary objectives, while keeping the backbone language model frozen.

The authors evaluate LAMP on eight question-answering and knowledge-editing benchmarks using two frozen language models. The results show improvements over conventional in-context injection and a modest F1 advantage over a LoRA-fine-tuned in-context baseline. Additional experiments investigate the contributions of the two injection pathways, their roles during reading and generation, robustness to distractors, retrieval from large corpora, and mid-generation evidence updates. Overall, the paper presents an interesting approach to external memory injection and provides useful empirical observations about the complementary roles of KV-based and residual-based information delivery.

Critical strengths

Interesting architectural design. The paper introduces two complementary injection pathways that deliver external evidence through the KV cache and upper-layer residual stream. The proposed architecture provides an interesting perspective on how compressed evidence can be incorporated into frozen language models.

Comprehensive evaluation across different scenarios. The experiments cover multiple question-answering and knowledge-editing benchmarks, together with investigations of distractor robustness, long-corpus retrieval, inference efficiency, and mid-generation evidence updates. The paper also reports useful negative results and limitations, including the degradation of single-memory compression under distractors.

Critical weaknesses

Overly complicated training objectives and insufficient ablation studies. The training objective in Eq. (1) combines ten loss terms with numerous additional hyperparameters, making the training procedure unnecessarily complex. Although Appendix E defines these objectives, the necessity and individual contributions of several regularization and auxiliary losses remain insufficiently justified. Moreover, the paper lacks systematic loss-level ablations, making it difficult to determine which training components are essential to the reported improvements.

Inconsistent accuracy improvements over the in-context baseline. Tables 1 and 14 show that LAMP achieves lower accuracy than the in-context baseline in several settings, despite its improvements in F1. For memory-based tasks, effectively utilizing retrieved evidence to generate accurate answers is essential. Therefore, higher F1 scores alone do not necessarily demonstrate superior memory utilization, particularly when answer accuracy remains lower than that of the baseline.

Insufficient evidence for the proposed functional separation between the two pathways. The evidence crossover experiment primarily examines the KV-only configuration, where answer recall is substantially reduced after disabling the residual pathway. However, this does not establish whether the KV memory preserves answer-specific information or whether the model simply fails to extract it without the residual pathway. Thus, the proposed functional separation between the two pathways remains insufficiently validated.

Questions for authors
  1. Interpretation of CE in the No-harm Loss: Could the authors clarify the intuitive meaning of cross-entropy (CE) in the no-harm loss and explain how this objective prevents memory injection from degrading the model's original performance?

  2. Ablation of Regularization Losses: Could the authors provide experimental results showing how model performance changes when the regularization losses are removed? In particular, how much do these losses contribute to the overall performance of LAMP?

  3. Further Validation of Pathway-Specific Roles: Could the authors provide a more systematic evidence crossover analysis comparing KV-only, residual-only, and full LAMP under both own-evidence and donor-evidence conditions? Combining these comparisons with phase-specific interventions would help clarify the respective contributions of the two pathways to evidence reading, memory representation, and answer extraction.

Reviewer_RXxjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAMP for supplying external evidence to frozen language models. One pathway supplies compressed key–value memory. The other reads token-level evidence features and updates the residual stream. The evaluation covers question answering and knowledge editing. Additional tests examine evidence updates during generation, passage retrieval, and compression costs.

Critical strengths
  1. The paper addresses a practical problem: updating a frozen language model with external evidence.
  2. The two pathways provide complementary benefits. Comparisons with single-pathway models of similar capacity support this result.
Critical weaknesses
  1. The experiments do not separate the effects of injection location and direct access to token-level features. The key–value pathway reads resampled latents, while the residual pathway reads token-level encoder features. The existing ablations support complementarity but do not isolate these two factors. A comparison using the same evidence representation would help test the proposed recall mechanism.
  2. Compared with In-context+LoRA, LAMP has higher token F1 but lower answer-string inclusion accuracy. Both metrics depend on answer length and wording. It remains unclear whether LAMP produces more factually correct answers. The comparison needs consistent answer extraction, additional correctness checks, and uncertainty estimates.
  3. The reported 85.2% correction rate is difficult to interpret without a clear success criterion. It is unclear whether negated mentions, multiple candidate answers, or contradictory claims are counted as successes. A separate check is needed to establish how often the model clearly adopts the new answer without retaining conflicting claims. The scoring criteria also need to be consistent across LAMP and the text baselines.
  4. Each backbone requires a separately trained projector, which adds model-specific training and deployment costs. In-context+LoRA provides a text-input baseline trained on the same data. LAMP's quality advantage over this baseline depends on the metric. The paper should clarify when these gains justify the extra training and the dedicated injection interface.
Questions for authors

Please refer to the Weaknesses section.

Reviewer_qMaeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces LAMP, a method for giving a frozen language model new evidence without putting the entire passage into its prompt. Its central idea is to combine a compact memory that guides the decoder with a separate reader that can recover detailed information from the passage. A frozen encoder produces token-level features, which are resampled into latent tokens and projected into the decoder's key-value cache. Sparse cross-attention adapters also read the original encoder features and update upper-layer hidden states. The added components are trained on 146K question-answer triples while the base model stays frozen.

Across two backbones, the combined model substantially outperforms either pathway alone. Against a textual LoRA baseline trained on the same data, it gains 1.52 F1 points but loses 0.84 accuracy points on Qwen3-VL-8B. Further experiments examine distractors, retrieval from larger corpora, inference costs, and evidence updates during generation.

Critical strengths
  1. On Qwen3-VL-8B, LAMP reaches 91.83% accuracy, compared with 64.17% for KV-only and 67.67% for residual-only training. Widening the KV-only model to the same 154M trainable parameters reaches 69.83% in Table 3. These results support the architectural combination and show that simply enlarging the tested KV-only model does not recover its benefit.

  2. The paper includes a textual LoRA baseline trained on the same 146K triples, retrained single-pathway models, inference knockouts, empty and incorrect evidence, and removal of CounterFact from training. These controls strengthen the evidence that the model uses the supplied information and that its two branches co-adapt. Reporting failures with distractors and short-passage cost disadvantages also helps identify the method's limits.

Critical weaknesses
  1. Injection location is confounded with access to detailed evidence. The full model's residual branch reads token-level encoder features, while the KV-only and input-layer controls receive resampled latents. Matching parameter count does not match the information available through those interfaces. The comparisons therefore support the combined architecture but do not establish that injection location causes the gain. Moreover, the main benchmarks have mean passage lengths of 13.2 to 183.6 tokens against a default budget of 256 latents, so they provide limited evidence for an explanation based on a tight compression budget.

  2. The advantage over a trained textual reader remains modest and incompletely characterized. In Table 11, LAMP gains 1.52 F1 points over In-context+LoRA but loses 0.84 accuracy points, without uncertainty for this comparison. Neither metric alone resolves differences in answer phrasing. The corpus-scaling experiment reads one retrieved passage rather than the whole corpus. Its matched short-passage comparison costs 43 ms and 140 MB for LAMP versus 38 ms and 131 MB for text in Table 16. Larger efficiency gains need answer quality measured on the same workload, especially for the multi-memory system used with distractors. These gaps leave the practical benefit less established than the architectural benefit.

Questions for authors
  1. How much of the dual-pathway gain remains when both branches receive the same compressed representation, or when a single-pathway reader receives the same token-level features and a matched parameter budget? This would distinguish the effect of injection location from the effect of providing another evidence interface.

  2. What are the full In-context+LoRA configuration and checkpoint-selection procedure, and the per-benchmark evaluation counts? Can you provide paired uncertainty for the 1.52-point F1 difference and an answer-level analysis of cases where accuracy and F1 disagree?

  3. Can you report answer quality, total latency, and memory for LAMP and the trained textual reader using identical retrieved passages on the long-evidence workload? Include encoding, retrieval, and all memories required by the multi-memory variant. This would establish where the proposed representation provides a practical benefit.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Halfway summary of the discussion

Update, 18 November. This summary was written earlier in the discussion, before any reviewer had replied. All three reviewers have since responded, and Reviewer qMae has raised the rating. Please see our closing summary, posted today, for where the submission now stands. The text below is kept as originally posted.

We thank the AC and the reviewers. We posted our responses and two new experiments about a week ago, and since we are more than halfway through the rebuttal period, we would like to summarise here where the discussion stands. We have also uploaded our revised paper based on the reviews, and all changes are highlighted in blue.

In short. All three reviews confirm the main result and rate their soundness concerns as "minor or fixable". Every concern is answered, most from the submitted paper and the rest with new experiments.


1. What all three reviewers agree on

RXxj: "The two pathways provide complementary benefits. Comparisons with single-pathway models of similar capacity support this result." The paper "addresses a practical problem: updating a frozen language model with external evidence".

qMae: The results "support the architectural combination and show that simply enlarging the tested KV-only model does not recover its benefit". The controls "strengthen the evidence that the model uses the supplied information and that its two branches co-adapt".

bSmi: An "interesting architectural design" that "provides an interesting perspective on how compressed evidence can be incorporated into frozen language models", with a "comprehensive evaluation across different scenarios".

No review questions the novelty of the design, names prior work that already does this, or disputes a number in the paper. We take this agreement as clear support for the value of the work.


2. The concerns are about interpretation and reporting, and all are resolved

None of them affect the results and strengths above. For each, we note what the paper already contained and what we added.

  • Injection site vs. token-level features (qMae, RXxj). Table 3 already compared two sites with the same latents: 64.17% in the KV cache and 46.33% at the input. We then trained both controls qMae proposed. They reach 81.67% and 73.17%, against 91.83% for LAMP (p < 10⁻⁵). The gain needs both parts, and the revision says so.
  • Accuracy vs. F1, and uncertainty (all). Appendix J already reported both metrics and where they disagree. The paired statistics we added show no significant difference between LAMP and In-context+LoRA (F1 +1.52 [−0.73, +3.78]). The "inconsistent" accuracy in Table 1 is 2 of 16 results, by 2 and 5 questions.
  • Functional separation (bSmi). Table 10 already included the full model, and Table 9 the phase-specific interventions the review asks for. Our new split crossover settles it: the answer follows the residual pathway (80.50% vs. 9.67%, p < 10⁻¹⁷).
  • Correction criterion (RXxj). Section 4.6 already reported how often an answer keeps the old target. Even if every such answer counts as a failure, LAMP corrects 54.6% of cases, against 2.5% and 0.0% for the text baselines.
  • Training objective (bSmi). The same losses without the two pathways reach only 46.33%, and the recipe transfers unchanged to a second backbone. We explained the no-harm term and gave the role and weight of every term.
  • Cost and practical benefit (qMae, RXxj). Table 16 already gave the cost on the same passages: close to text when passages are short, and lower as they grow (Figure 4). We also listed when LAMP is the better choice and when plain text is enough.
  • Configuration and counts (qMae). We have provided the full In-context+LoRA configuration and the number of questions per benchmark.

3. Summary

  • The reviews confirm the central result, rate their own concerns as minor or fixable, and identify no flaw in the method or the experiments.
  • Many of the points raised were already answered in the submitted paper. For the rest, we ran new experiments.
  • Reviewer RXxj gives the lowest rating, 1 (clear rejection). Yet the same review says that our comparisons support the main result, and it asks us no questions. None of its four weaknesses disputes a result in the paper. The review itself rates them as minor or fixable, and the submitted paper already answers much of what they ask: the same-latent site comparison (Table 3), the old-target rates (Section 4.6) and the metric analysis (Appendix J). We see nothing in this review that supports a rejection.

We do not think the current ratings follow from what the reviews themselves say, and none has been revisited since our responses. We respectfully ask the AC to weigh the reviews by their content and by this discussion.

AuthorsOpenReview ↗

Closing summary of the discussion

As the discussion closes today, here is our summary.

In short. All three reviewers confirm the main result, and every concern has been answered. qMae raised the rating to 3 and soundness to "no substantive concerns". bSmi set two conditions for reassessment; we delivered both on 16 November. RXxj alone keeps a 1, in a review that calls its own concerns "minor or fixable" and the main result supported. RXxj replied once, mostly with points the other two reviewers had just posted, and left our requests for a reason unanswered.

1. What the reviewers agree on

  • Problem: "a practical problem: updating a frozen language model with external evidence" (RXxj).
  • Main result: "Comparisons with single-pathway models of similar capacity support this result" (RXxj). "simply enlarging the tested KV-only model does not recover its benefit" (qMae).
  • Design and evaluation: "Interesting architectural design", "Comprehensive evaluation across different scenarios" (bSmi).
  • After our replies: "my concerns are mostly addressed" (qMae). "substantially alleviated my concerns" (bSmi). The new controls "strengthen the evidence for the dual-pathway design" (RXxj).

2. Why the paper matters

The usual way to give a frozen LLM new information is to put the passage in its prompt. Recent latent methods write a compressed memory into the attention cache, but lose accuracy. LAMP instead uses two pathways with different jobs: a cache memory gives every layer the gist, and upper-layer adapters restore the token-level detail that compression loses.

  • It beats the prompt and other latent methods. It leads the prompt by 7.1 and 9.7 points on two LLMs, and MAC and KBLaM by 37.8 and 49.6.
  • It does what a prompt cannot. A correction injected mid-answer takes effect in that answer in 85.2% of cases, against at most 2.5% for prompt-based readers.
  • It matches or beats fine-tuning. LAMP leaves the LLM frozen, yet is level with a fine-tuned LLM (F1 93.18 against 91.66) and ahead in five other tests. Fine-tuning also has a cost: in three of four robustness tests it is worse than the frozen LLM.
  • The gain comes from the two pathways together. At the full model's size, the cache memory alone reaches 69.83% and the adapters alone 81.67%, both below the prompt (84.7%). Together they reach 91.83%.

3. Reviewer qMae

We thank Reviewer qMae sincerely. qMae proposed two new controls, asked for the paired statistics, and pressed us on the practical advantage. qMae then wrote: "my concerns are mostly addressed so I will increase the score to 3".

4. Reviewer bSmi: the stated condition for reassessment has been met

In bSmi's words on 15 November:

  • "My remaining concerns focus on the training objective and its implementation."
  • The rating stays at 2 "for now", and a core-objective comparison and the objective code "would materially inform my reassessment".

We delivered both in 24 hours. With only the three core loss terms, LAMP still reaches 82.17%, and the full objective adds 9.7 points. The code is in the anonymous repository. bSmi replied only in the last hours, acknowledged "the complementary benefits of the two-pathway design itself", and kept the 2 and moved the goalposts, and also misread the paper (In-context+LoRA is called omitted, but is in Tables 2 and 4); please see our full reply there. The one concern bSmi asked us to test was tested, and the result supports the objective. We ask the Area Chair to weigh the rating against that record.

5. Reviewer RXxj's rating

Every concern RXxj raised was also raised by qMae or bSmi, and each was answered:

RXxj’s concernShared withOutcome
Injection site vs. token-level featuresqMaeTwo new controls. RXxj: "strengthen the evidence".
Accuracy vs. F1, uncertaintyqMae, bSmiPaired statistics. bSmi: "substantially alleviated".
Practical advantage over fine-tuningqMaeAnswered. qMae raised the rating.
Correction criterionqMaeStrict rate added. RXxj: "is helpful".
Training objectivebSmiExperiment run, code released.

RXxj's one reply came two days before the close. Three of its four points had just been posted by the other two reviewers, one almost word for word, and it ends: "Overall, I do not think the contribution and supporting evidence are strong enough for ICLR." That is a conclusion, not a reason. We asked twice for the specific flaw that makes this paper a clear rejection, and no answer came.

The ICLR Reviewer Guide asks for a “substantive” review, a reviewer “actively engaged”, and a statement of “what did/didn’t change your recommendation”, and RXxj did not meet these.

We cannot answer an impression that is never explained. We ask the Area Chair to weigh this rating by what the review and reply contain.

Finally, we thank the reviewers whose questions made this paper stronger, and the Area Chair for the time and care this takes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.