acceptodds
Under review as a conference paper at ICLR 2027

Memory Outlives the Model: Transplanting In-Attention Memory States Across LLM Backbones

Abstract

Latent memory states inherit the interfaces of the models that write them. We separate the two. Agents now outlive the models beneath them, and their accumulated memories must travel across checkpoints, as text already does through a shared symbolic format. In a delta-rule memory, the accumulated state is a small r × r matrix per layer, independent of the writer’s hidden size, tokenizer, and depth; only the read and write projections are model-specific. The state is a portable file. We present the first cross-backbone transplant of a fixed-size episodic in-attention state: a frozen writer encodes an episode into its state once, and a frozen reader from another family decodes it inside its own attention through a ∼3M-parameter read head. With the episode absent from the prompt, the state alone significantly lifts five readers from four families (3B–32B) on 21 of 25 reader–benchmark pairs. The state is also a format. Independent writers store the same episode in unrelated per-layer coordinates, so a head fit to one writer reads another as noise. Each writer sets its axes and scale alone from its mean-state SVD; one frozen head per reader then reads writers it never saw (16/16 pairs; 44/50 with a single universal head). Added to retrieval, the state improves every reader on 121K-token histories and adds +4 to +21 points on top of the gold evidence sessions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cross-Model Memory Transfer, Latent Memory Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2qVXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Building on δ-mem, this paper studies how delta-rule associative memory states can be transferred from a writer model to readers with different backbones. The authors use a lightweight read head for each reader and introduce writer-side state canonicalization to support transfer across writers. Experiments show that the transferred states can improve question-answering performance, even when the source text is absent from the reader’s prompt.

Critical strengths
  1. The paper is well-written and technically sound.
  2. The motivation is clear.
Critical weaknesses
  1. What's the stratege to set the relative-depth mapping in Eq. (6) across different backbones? A small ablation with alternative layer mappings would help justify this choice.

  2. Paper only talk the condition that once-writter write and different-readers read. The paper handles multi-session histories, but it is unclear how independently written memory states would be retrieved, combined (or Merged), or updated as new conversations arrive.

  3. How does the reader handle conflicts between memory and the current context? Testing outdated memories against explicit corrections in the prompt would help assess the risk of factual errors or hallucinations.

  4. Beyond the alignment results in Appendix D, could the authors report how episode states vary in the coordinates defined by $\boldsymbol{M}_\ell$, and how stable these coordinates are across calibration samples?

  5. Paper treates the KV-states as associative memory states to preserve, which is similar to LoRA adapation [1] (or HyperNet adaptation) which is also considered as memory adapted for each tasks. Could author explain some advantages for KV-states writting-and-reading?

[1] Mind Lab. MinT: Managed Infrastructure for Training and Serving Millions of LLMs. arXiv:2605.13779, 2026. https://arxiv.org/abs/2605.13779

Questions for authors

See above

Reviewer_hdeqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work is built on δ\delta-mem [1], a (delta-rule state) side-network adapter for fine-tuning. This work learns a frozen writer (for model A) to create a small delta-rule state. Then on model B, this work learns a reader to read the state. per-writer SVD is applied for canonical coordinates.

[1] Lei, Jingdi, et al. "$\delta $-mem: Efficient Online Memory for Large Language Models." arXiv preprint arXiv:2605.12357 (2026).

Critical strengths
  • The experiments are broad, including several model families (3B-32B), 5 benchmarks, etc.
  • This work use careful statistics, including confidence interval, seeds, etc.
Critical weaknesses
  • The novelty is limited. The writer is \delta-mem, a side-network adapter. The reader finetunes around a fixed state representation. SVD for "canonical coordinates". The state transfer across model is expected by construction. The state S is a gated sum of outer products, a linear associative memory. Then reader is also linear in S. Model stitching [2,3] already shows that simple linear maps connect independently trained models.

  • The writers in this work share same training data. Thus, the cross-writer agreement may therefore reflect shared training rather than a universal format.

[2]Lenc, Karel, and Andrea Vedaldi. "Understanding image representations by measuring their equivariance and equivalence." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[3] Bansal, Yamini, Preetum Nakkiran, and Boaz Barak. "Revisiting model stitching to compare neural representations." Advances in neural information processing systems 34 (2021): 225-236.

Questions for authors
  • How large are the gains over a plain rank-8 LoRA on the reader (same projections, comparable parameter count), trained on exactly the same QA pairs and prompts used to fit the read head?
1 comment
Reviewer_hdeqOpenReview ↗

Reviewer response

Thanks for your response!

W1: Novelty. This is the main problem. The rebuttal never says what is conceptually new or surprising. The title and introduction are ambitious, but the content is neither new nor surprising.

W2: Shared training. Not addressed. If writers trained on the same data agree only after per-writer SVD, that is exactly what my explanation predicts: the same structure up to a change of basis. Table 4c varies the calibration text, not the training data.

Q1: LoRA. Answered in letter, not in spirit. LoRA never sees the random facts, so its 3–5% was fixed by the design. The question is to know how much of the benchmark gain comes from QA fine-tuning, and the rebuttal explicitly declines to measure that.

Reviewer_uNkGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for transferring a fixed size memory state written by a frozen language model so it can be read by a different language model independently of the family, size, depth or tokenizer. The memory is taken from a delta-rule adapter that compresses an episode into a rxr matrix per attention layer which is made readable by other backbones through a parameter-free depth remap and a thin read head trained once per reader. Their main contribution is a protocol for reading this matrix in a different frozen model through a small read head trained once per reader. They also introduce a canonicalization step that puts each writers matrix into shared coordinates, so one head can read writers it was never trained on. The approach is demonstrated with experiments on five readers from four model families across conversational memory and multi-hop QA benchmarks.

Critical strengths

The evaluation is broad, covering five readers from four model families and five benchmarks, with random-state and random-head controls in every regime that rule out the head alone as the source of the gains. The canonicalization step is simple and well motivated, with a clear explanation of why raw states fail to transfer and ablations showing what the convention needs to fix.

Critical weaknesses

A key claim of the paper is that the matrix size is independent of episode length, which is true of the stored state, but writing still requires the writer to run over the whole episode, so episode length is bounded by the writer’s context window. The motivating use case as stated in the introduction is agents running for weeks or months, at least to the point where you would switch models, whose histories could exceed this.

Every read head is trained on question-answer pairs from the benchmarks it is later evaluated on, either per benchmark or pooled across all five (§3.5, Appendix F.3). The evaluation episodes are held out, but the question style and domain are not, so the results do not show whether a head generalizes to a kind of episode or question it was not trained on. This matters for the paper’s motivation, since a deployed agent would not have benchmark-style QA pairs for its own history.

Questions for authors

The paper motivates the state as memory that survives model upgrades, but in the experiments the state is written once and only read afterward. In the agent setting described in the introduction, the new model would also need to keep adding to the memory. Can the authors comment on whether a new writer could continue from a canonicalized state produced by a previous writer, or whether the full history would need to be rewritten?

LoCoMo evaluates the matrix alone, retrieval alone, and their combination (Table 22), whereas LongMemEval-full evaluates only the latter two (Table 23). Given this, the matrix’s standalone effectiveness is tested only up to 26K tokens, leaving its effectiveness at 121K unmeasured. Could authors add a state-only condition to Table 23 to address this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.