acceptodds
Under review as a conference paper at ICLR 2027

MemoryAthena: Adaptive Routing over Latent and Generated Memories

Abstract

Learned-memory methods store information in an explicit table and consume it through a separate reader, allowing addressing, storage, and reading to be modified independently. Prior work on cross-model memory transfer exploits this separation to reuse learned memory across frozen backbones through an adapted reader, while the representation consumed by the model still originates from stored memory. This raises a natural question: must useful memory always be retrieved from storage, or can it also be generated? We investigate this question with MemoryAthena, a memory interface with three pathways: direct Engram retrieval (E), generation from retrieved Engram cues (GE), and generation from causal backbone states without consulting the memory table (GH). Generated memory is not uniformly better than direct retrieval: it can complement E in one context but interfere with it in another. MemoryAthena therefore treats E as an explicit anchor and learns when a generated representation should intervene. With the backbone, memory, generators, and readers frozen, a lightweight causal routing head is trained from counterfactual future-token likelihood advantages of GE and GH relative to E. At inference time, an admitted candidate modifies the E residual through bounded interpolation, while rejection recovers the direct pathway exactly. On question answering, MemoryAthena raises the five-task average from 37.65 to 39.28 over the direct pathway of the same checkpoint, while the six-task general-NLP average increases from 76.73 to 79.13. The complete memory-side system contains approximately 201M parameters, excluding the frozen backbone. Further analyses show that the utility of E, GE, and GH varies across tasks and inputs, while gold-label oracles reveal additional complementarity among the three pathways. These results support generated memory as a selective correction to direct retrieval rather than a universal replacement, and highlight routing when, which, and how strongly to intervene as the central challenge. Code is https://anonymous.4open.science/r/ATHENA-0CFB/README.md

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZCbLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemoryAthena studies whether memory consumed by a frozen LM must be retrieved or can be generated. Building on an addressing/storage/reading decomposition (Li et al. 2026) over the Engram addressable memory (Cheng et al. 2026), it defines three memory-side pathways over a frozen backbone (Mistral-7B-v0.3, memory injected at layers 2 and 10): direct Engram retrieval (E), generation from retrieved Engram cues (GE), and generation from causal hidden states (GH). The key claim is that generated memory is conditionally useful, not a uniform replacement, so the method keeps E as an anchor and admits a generated candidate via bounded interpolation r = e + α(g − e) only when a lightweight causal routing head predicts positive E-relative advantage and sufficient confidence (Algorithm 1). The router (534,924 params) is trained label-free by distilling counterfactual future-token likelihood advantages aggregated over horizons {1,4,8,16,32} under teacher forcing, with everything else frozen.

Validation: five QA benchmarks (NQ/WebQA/TriviaQA/TruthfulQA/HotpotQA) and six NLP classification tasks (dCPMI scoring). On the same checkpoint, the QA five-task average rises 37.65→39.28 and the NLP six-task average 76.73→79.13 over E-only. Ablations cover from-scratch vs pretrained memory, no-gate, permuted keys, parameter-matched FFN, affine stitch, and a random-router control; analyses include gold-label source oracles, routing-mass statistics, GPT-2 scaling, HaluEval, Mistral→Llama transfer, and an inference microbenchmark. Code is provided (anonymous 4open.science link).

Critical strengths
  1. Correct problem framing with a safe mechanism. Treating generated memory as a candidate correction anchored to a strong direct pathway, with bounded interpolation and exact fallback, is both conceptually apt and operationally robust: if the router is wrong about admission, the worst case at a rejected site is the E baseline. The random-router control (39.28→31.67, −7.61) convincingly shows the gains come from learned selectivity, not merely from exposing multiple pathways — this is the most decision-relevant ablation.
  2. Label-free routing supervision. Distilling token-level E-relative advantages from future-token likelihood (Eqs. 8, 17–19) separates constructing candidate memories from selecting them, without downstream labels, and the router is parameter-light (0.5M). This is a genuinely reusable recipe and is supported by released code.
Critical weaknesses
  1. Small, fragile, and partly confounded gains. The flagship QA improvement is +1.63 average on a same-checkpoint comparison, with no variance reported; it leans heavily on NQ (+4.74) while TruthfulQA (+0.49) and HotpotQA (+0.30) barely move and EM drops on two tasks. The NLP average depends on a test-tuned Yahoo threshold. Transfer (Mistral→Llama) and HaluEval results are explicitly "diagnostics" (e.g. HaluEval summarization +24.56 but QA −4.45, with the authors noting confusion-matrix/class-balance checks are needed). For a method whose premise is "conditional, selective benefit," the evidence that the net benefit is real and stable is weaker than it needs to be.
  2. Cost/benefit and headroom. MemoryAthena is not a zero-overhead router: GH requires a separate clean-backbone forward pass, giving 1.40–1.88× latency and +33–63% peak memory over E-only (Figure 10), while the deployed router captures only part of the available complementarity (large oracle gap, Figure 2b). So the method pays a real inference tax for a small, partly-confounded gain and leaves most of its own stated upside on the table — which undercuts the case that this is the right instantiation.
Questions for authors
  1. Please report seeds/CIs for Table 1 and Table 2. Are the +1.63 QA and +2.40 NLP averages outside run-to-run variation?
  2. Yahoo uses τ = 1 selected on test. What is the NLP six-task average if τ is fixed to the default (τ = 0) or selected on a held-out split for all tasks? Does the 76.73→79.13 gain survive?
  3. TriviaQA/HotpotQA show F1↑ but EM↓. Can you characterize what changes in the generated answers (length? hedging? added tokens)? Does routing improve answer correctness or mainly token overlap?
  4. Given the contamination caveats (Amazon/IMDB vs review benchmarks; non-disjoint CC-News), can you provide a decontamination or n-gram-overlap audit for the NLP tasks?
  5. Since the 3-source oracle beats the router by up to 9.1 F1, what is the ceiling of the current causal feature set — i.e. how much of the gap is irreducible without future-token information?
  6. How does MemoryAthena relate to Frozen Memory Is Not Enough: Rethinking External Memory as Extraction (arXiv:2608.17050), which makes a similar "retrieved memory need not be used as-is" argument? Please position the generated-memory + E-anchored-routing contribution against it explicitly.
Reviewer_AFagOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper looks into generating learned memory representations instead of retrieving them from memory stores. MemoryAthena proposes three memory pathways: direct Engram retrieval (E), generation from retrieved Engram cues (GE), and generation from backbone hidden states (GH). The approach maintains E as a reference and trains a light-weight router to predict the advantages of GE and GH relative to E based on differences in future-token likelihoods between frozen pathways. During inference, the router selectively combines a generated representation with E via interpolation. Experiments on question answering and general NLP tasks demonstrate performance gains over the E pathway.

Critical strengths
  1. Training a router with likelihood differences between frozen pathways is a sound way to get supervision signals without task-specific labels during training.
  2. Maintaining E as a baseline is an appropriate design since neither GE nor GH consistently performs better than direct E retrieval.
  3. This paper evaluates different routing and fusion techniques with the same checkpoint. According to Table 1, MemoryAthena improves the five-task QA average from 37.65 to 39.28 over E and outperforms the other routing methods.
Critical weaknesses
  1. The difference between generated memory and existing representation transformation methods is not evident. GE performs the transformation of memories that were retrieved, whereas GH produces the representations based on backbone hidden states without using external memory. In addition, it is not obvious how those pathways may differ from some more sophisticated memory readers and adapter modules. The authors do not provide any comparisons with these alternatives within the same parameter budget.
  2. There is probably some mismatch between how the router is trained and how it is used in practice. The advantage targets are computed based on comparisons of the whole E, GE, and GH pathways; however, the router makes local decisions at the token and injection-layer levels. Appendix 8.3 states that the advantage of the whole pathway cannot be assigned to one particular layer. No analysis of the correlation between these targets and the gains from local interventions is provided.
  3. In the general NLP evaluation, the routing threshold for Yahoo Answers is chosen based on the downstream test performance, as noted in Appendix 9.1. With the default threshold value, the Yahoo accuracy is 45.90 instead of the stated 57.43. Using the default threshold would reduce the six-task average from 79.13 to approximately 77.21, leaving only a 0.48-point improvement over E.
  4. The ablation studies do not shed sufficient light on the source of improvements. As can be seen from Table 3, shuffling Engram keys decreases the QA average by just 0.70 points, and training the memory module from scratch works better than using pretrained memory. GH also accounts for the majority of effective interpolation weights across multiple tasks (see Fig. 2). Thus, the contributions of memory retrieval and the two new generated pathways remain unclear.
  5. The improvements with respect to the direct E pathway are moderate in some QA tasks such as TruthfulQA and HotpotQA. The generated pathways add extra inference costs (see Appendix 18). The paper does not provide inference cost estimates for the core Mistral-7B experiment or uncertainty estimates for downstream improvements.
Questions for authors
  1. In what way is the "Permuted Engram keys" baseline applied in Table 3? Are the keys permuted only during inference, while the other components are left unchanged?
  2. Can the authors provide E+GE and E+GH results with the proposed E-anchored routing approach, instead of the separately trained baselines in Appendix 10? This would help assess the impact of each generated pathway separately.
  3. Did the authors explore the correlation between the predicted E-relative advantages and the actual effects of local interventions at individual injection layers?
4 comments
Reviewer_AFagOpenReview ↗

Response to the Rebuttal

Thank you for addressing my previous concerns. However, I still have one major question: why does the paper choose to study these three pathways, E, GE, and GH?

I understand that Li et al. (2026) provides the addressing/memory/reading decomposition, and that the paper considers generation from either retrieved memory cues or backbone hidden states. However, I still do not see how this leads specifically to E, GE, and GH. The motivation for choosing these particular pathways is not sufficiently explained.

For example, following the same logic, why not also consider a pathway that directly transforms backbone hidden states into residuals using a learned reader or adapter, without a separate generator? What makes E, GE, and GH a meaningful set of memory pathways rather than three particular architectural choices? I do not expect the paper to cover every possible design, but I still find it difficult to understand why these three pathways were selected instead of more or fewer alternatives.

If this concern can be adequately addressed, I would consider raising my rating to 3.

Thanks for your feedback, actually, the motivation comes from a classical view of human memory[1]. The hippocampus can be viewed as storing sparse, indexed traces, while the cerebral cortex reconstructs much richer representations from a relatively small number of cues. We use this analogy in our architecture. Engram behaves somewhat like the hippocampus: it provides fast and addressable cues. But instead of directly consuming those cues, perhaps a neural generator can reconstruct a richer latent memory representation, it also like human dreaming or recall according to the cues. In this view, retrieval tells us where to look, while generation tells us what richer representation should be reconstructed from the retrieved evidence. This motivates MemoryAthena, we also uploaded the new version and add some introductions in Figure1. Also for the figure3(b) and table1 router policy results, you can see that just using one pathway E, two pathway E+GE can not achieve to the best performance, if we use three pathways we can get more better performance.

Thanks for your residual idea, I’ve thought about your residual architecture before, now we replaced GH's generator and reader with a learned adapter and directly transforms backbone hidden states into residuals:

  • Input: the same 3-token causal window of clean hidden states.
  • Architecture: RMSNorm → Linear(3·4096 → 256) → SiLU → Linear(256 → 4096).
  • Gate and placement: the same sigmoid gate as GH, at the same injection layers.
  • Size: 8.4M parameters.
  • Training: the same objective and corpus (20M Wikipedia tokens). The backbone, Engram table, E and GE stay frozen.
Pathway (alone, Mistral-7B)Val. PPL ↓QA5 avg. F1
E8.5737.74
GE7.1535.39
GH (generator + reader)7.0933.21
H-direct (adapter, no generator)7.0437.77

On its own, H-direct is at least as good as GH. A separate generator is therefore not needed for single-pathway quality, but its exact-match is lower than E's, however (NQ EM 16.5 vs. 20.4), which indicates longer, less precise answers. We retrained the counterfactual router over {E, GE, H-direct} with the original protocol.

Router candidatesNQWebQATriviaQATruthfulQAHotpotQAAvg.
E only28.4133.3969.3831.4626.0437.74
{E, GE, H-direct}29.5932.8771.4631.2527.1838.47
{E, GE, GH} (MemoryAthena)33.0234.6070.6831.7426.3439.28

A pathway that is stronger in isolation is therefore not necessarily a better candidate. It shows that router design is also a key factors to improve the performance.

[1] The hippocampal memory indexing theory. [J]. Behavioral neuroscience, 1986, 100(2): 147.

Reviewer_AFagOpenReview ↗

Response to the Rebuttal

Thank you for the additional experiments. These results do strengthen the paper's argument, and I will raise my rating to 3.

The neuroscience analogy helps me understand the motivation for designing GE. However, I still have a minor concern about why E, GE, and GH were chosen specifically. The current presentation still makes these three pathways seem somewhat arbitrary, which is why I asked why another pathway was not included. I don't think this necessarily requires more experiments, but rather a clearer rationale for these choices to make the paper's logic easier to follow.

Thank you again to the author for the response and feedback. Your question has prompted me to think more deeply about the essence of the issue: why do GE, GH, and E work? My initial thought, based on cognitive science, is that E resembles an index-based retrieval mechanism, while generative memory can associate memories linked through the index. To verify whether our architecture reaches similar conclusions, we conducted some case studies,

Case 1: Associative reconstruction (GE). When asked who played young Clark Kent in the 1978 Superman film, E predicts Christopher Reeve, whereas GE correctly recovers Jeff East, which is also selected by the router. Similarly, for the character Alice Tinker in The Vicar of Dibley, E predicts Dawn French, while GE and the router correctly identify Emma Chambers. These examples suggest that retrieval-guided generation can recover factual associations that are not directly recovered through E.

Case 2: Preserving precise factual retrieval (E). When asked when skiing halfpipe became an Olympic event, E correctly predicts 2014, whereas GE and GH produce incorrect years. The router retains the correct answer. This illustrates the complementary value of direct retrieval when generative corrections are unreliable.

Case 3: Context-driven reconstruction (GH). When asked when Michigan last defeated Ohio State, E and GE predict 2003, whereas GH correctly identifies 2011 according to the benchmark's historical reference. The router also produces 2011. Unlike GE, GH generates its correction directly from the backbone's causal states without relying on retrieved Engram cues. This example illustrates how context-driven generation can provide complementary information when direct retrieval and retrieval-guided generation fail to recover the correct answer.

Thus, E, GE, and GH are not arbitrarily chosen components, but three complementary mechanisms within our frozen-backbone memory correction framework: direct retrieval, retrieval-guided reconstruction, and context-guided reconstruction. The router is designed to integrate their potentially complementary predictions rather than assume that any single mechanism is always preferable.

Finally, as the rebuttal is ending, i don't have too much time to exchange ideas. However, your feedback has prompted me to think more deeply on our approach and has inspired directions for our future research, including how to use residual architectures and how to further improve E, GE, and GH. Once again, thank you for your feedback. By the way, the rest experiments also finished, i have already updated it.

Reviewer_jPB5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemoryAthena, a memory interface operating on a frozen LLM. It defines three pathways: direct Engram retrieval (E), generation conditioned on retrieved cues (GE), and generation from the backbone’s clean hidden states without table access (GH). Rather than routing symmetrically over the three, E is treated as an anchor: a causal head predicts the E-relative advantage and confidence of GE/GH, admits a candidate only when both exceed thresholds, applies bounded interpolation r = e + α(g − e), and otherwise falls back to E. The router is trained without downstream labels, using next-token likelihood differences between frozen endpoints (counterfactual advantage). On Mistral-7B, the paper reports improvements over the same-checkpoint E pathway from 37.65 to 39.28 on the five-task QA average and from 76.73 to 79.13 on the six-task classification average. Ablations, oracle analysis, GPT-2 scaling, and inference-cost measurements are provided.

Critical strengths
  • S1. Clear question and good motivation: must a parametric memory representation be read from storage, or can it be generated? The GH results suggest the stored table contributes less than one might expect.
  • S2. The E-anchored asymmetric routing is a sensible design and is directly supported by controlled experiments: it is the only inference rule that improves over E only (37.65 → 39.28), whereas symmetric hard/soft routing (35.83 / 28.62) and a random router (31.67) all fall below it.
  • S3. Label-free router training via counterfactual advantage distillation is clean and separates “how to construct memory” from “when to use it.”
  • S4. The oracle analysis honestly quantifies the remaining headroom (3–9 points) and correctly identifies routing, rather than candidate quality, as the bottleneck.
  • S5. Inference overhead (1.4–1.9× latency, 33–63% memory) is reported.
Critical weaknesses

I found it difficult to decide on a score. My main concerns are W2 and W5. I hope to see these points addressed in the author response.

  • W1. No confidence intervals, seeds, or significance tests. Some QA gains are small (HotpotQA +0.30, TruthfulQA +0.49).
  • W2. The Yahoo threshold τ = 1 was selected using test performance (Appendix. 9.1). The six-task average therefore includes a test-tuned entry. Please report τ sweeps on at least NQ and SST2. Is τ = 0 optimal there, or merely the default?
  • W3. Training from scratch (40.16) outperforms the pretrained-memory configuration (39.28), which undermines the memory-transfer framing that motivates the setup. This is reported but not explained.
  • W4. On TriviaQA and HotpotQA, F1 increases while Exact Match (EM) decreases (Figure 4). This pattern is consistent with longer rather than more accurate answers, since F1 measures token overlap. Can you report the mean generated answer length?
  • W5. Cross-backbone NLP transfer (Table 11) is at chance level on SST2/Yahoo and lacks a bare-Llama baseline. In addition, the +24.5-point HaluEval summarization gain is flagged by the authors themselves as possibly due to response bias.
  • W6. The training corpus includes Amazon and IMDB reviews, overlapping in domain with SST2/MR/RT, yet no contamination audit is discussed.
Questions for authors

Please see Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.