acceptodds
Under review as a conference paper at ICLR 2027

TransMem: Transforming Hidden States into Memory for Large Language Models

11 views

Abstract

Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose TransMem, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58–16.88 on LoCoMo and 10.20–13.03 on HotpotQA, while improving the average MemoryAgentBench score from 28.95% to 40.00%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based Agents, (Large) Language Models, AI Agent Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1vyLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TransMem, an inference-time parametric memory module that reuses hidden states a frozen LLM has already computed during prefill. The context is split into segments (four by default) and the final hidden state of each segment serves as a sparse memory feature. During answer generation only, a small transformer block with a learned projection and a gating network maps these features to a memory shift that residually corrects hidden states at the last four layers. Training uses evidence-conditioned self-distillation: a teacher sharing the same frozen backbone sees only the gold evidence and the question, while the student sees the full context, and the module is optimized with a forward KL at the output distribution, updating only the memory modules.

Critical strengths
  1. The method is genuinely lightweight with context-independent overhead. With only four segment-final hidden states as memory, modules inserted into the last four layers and activated only during generation, compute and latency stay nearly constant as context grows (Figure 3), in contrast to the roughly linear cost of δ-Mem. Gains are consistent across three backbones and three benchmarks, so the improvement is not tied to one architecture.
  2. The ECSD objective is well motivated and the transfer results support the central claim. Distilling an evidence-only teacher into a full-context student provides token-level evidence-conditioned supervision, and Table 4 (SFT 48.82, GRPO 52.42, OPD 52.80, ECSD 54.11) isolates the contribution of the objective. Training only on HotpotQA and evaluating on LoCoMo and MemoryAgentBench supports the claim that the module learns a general memory capability rather than task knowledge.
Critical weaknesses
  1. Baseline comparability and statistical rigor are insufficient. The training corpora and setups of the parametric baselines in Tables 1 and 2 (δ\delta-Mem, MLP Memory, Context2LoRA, MemGen) are not specified; if they follow their original configurations, the comparison confounds training-data effects with architectural ones. No multi-seed variance is reported, yet differences of 0.2 to 1 F1 in Tables 5 and 7 are used to support conclusions. On MemoryAgentBench (Table 2), TransMem trails δ-Mem on Accurate Retrieval (41.60 vs. 44.40) and MemGen on Long-Range Understanding (36.19 vs. 44.30); the aggregate lead rests on Test-Time Learning and Selective Forgetting, so precise retrieval ability is not established.
  2. The memory mechanism itself lacks direct evidence. Four memory states must stand for contexts up to 256K tokens, yet the paper offers no analysis of their capacity. Table 13 shows performance is nearly invariant to the number of segments B (three of five subtask scores are identical from B = 4 to 32), which is more consistent with the memory states providing a coarse global signal than with retrievable long-range evidence. Combined with Table 4, where plain SFT already recovers +8.0 of the +13.3 total gain and ECSD adds about +5.3, a substantial part of the improvement may come from generic output correction learned by training the module rather than from recovering long-range information. Without probing or perturbation experiments on H_M (e.g., masking or shuffling the segments containing gold evidence), the core claim is under-supported.
Questions for authors
  1. Were the parametric baselines in Tables 1 and 2 retrained on identical data, or are they original checkpoints? Please specify the training corpus and budget per baseline so that training-data effects can be ruled out.
  2. Please explain the insensitivity to B in Table 13 and provide analysis (e.g., masking or shuffling the segments that contain gold evidence, probing H_M) showing that the four memory states carry retrievable long-range information.
  3. Are the main results single runs? Please report mean and standard deviation over at least three seeds, especially for the differences below 1 F1 in Tables 5 and 7.
Reviewer_e8G5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TransMem, a parametric memory module added to the last layers of a frozen LLM. During answer generation the module reads a few historical hidden states, one per context segment, together with the answer-position states, turns them into a memory shift and adds it to the current hidden state through a learned gate, so the context is not re-encoded. The module is trained by evidence-conditioned self-distillation, in which the same frozen backbone given only the gold evidence and the question acts as teacher for a student that sees the full context. Trained on HotpotQA, TransMem improves three backbones on LoCoMo and HotpotQA and Qwen3-4B on MemoryAgentBench, with larger gains over the backbone alone than over the strongest parametric baseline.

Critical strengths
  1. The paper poses a distinct test-time question, namely how a frozen model can make better use of hidden states it has already computed for a long context, instead of storing text or re-encoding the context. Reading a handful of prefill states and acting only during answer generation keeps the module's cost independent of context length (Section 3.3, Figure 3), which is the practical reason to care about this framing.
  2. Evidence-conditioned self-distillation is a clean training signal, because the teacher is the same frozen backbone given the gold evidence, so no separate teacher model is needed. The paper tests this objective against supervised fine-tuning, on-policy distillation and GRPO on the same module (Table 4), which is the right way to justify a training objective.
Critical weaknesses
  1. The experiments do not isolate the contribution of the sparse historical hidden states. No ablation removes or randomizes the memory states while keeping the rest of the module, since Table 3 removes only the gate and the projection and Appendix A.2 changes only the layer the module reads from. Table 13 changes the number of memory states from 4 to 32 only at inference, on a checkpoint trained with 4, and its nearly unchanged scores fit robustness as well as little use of the memory states. The same module trained with plain supervised fine-tuning reaches 48.82 F1 on LoCoMo (Table 4), above every Qwen3-4B baseline in Table 1, but a control of similar size trained without the memory states is needed to tell how much of the gain comes from them. The abstract concludes that sparse historical hidden states are "an effective and efficient memory substrate", so the missing ablation of the memory states is the main reason for my recommendation.
  2. The comparison with the parametric baselines does not document matched training data and parameter budgets, so the size of the advantage over them is uncertain. Table 14 lists TransMem's training data alongside the training data reported in the original papers of δ-Mem, MLP Memory and MemGen, in units that differ, and leaves out Context2LoRA, and the paper does not say how these baselines were trained or sized for Tables 1 and 2. No parameter count is given for the TransMem modules, which would total several hundred million parameters on Qwen3-4B if each added transformer layer matches a backbone layer in size. This uncertainty weakens the comparison with the parametric baselines in Tables 1 and 2.
Questions for authors
  1. What does the module score without the memory states, so that it reads only the answer-position states, or with random vectors in their place, and what does a LoRA of similar size on the last four layers reach when trained with the same evidence-conditioned self-distillation on the data TransMem used? These comparisons would help isolate the contribution of the memory states.
  2. Section 4.1 says TransMem is trained only on the HotpotQA training set, while Appendix A.1 says all its variants use a QASPER and HotpotQA training set and its β = 0 row in Table 9 repeats the main result of 54.11. Which is correct, and were the parametric baselines retrained on the data TransMem used?
  3. How many parameters do the TransMem modules add for each backbone, and how do end-to-end latency and GPU memory compare with the frozen backbone alone and with δ-Mem when the prefill is included?
  4. The paper reports no variation across runs for any score, which matters most for the lead of 1.15 points over δ-Mem on the MemoryAgentBench average. How many runs underlie Tables 1 and 2, and what is the variation across runs?
  5. LongMem (NeurIPS 2023) also keeps the backbone frozen and trains a residual side network with a learned gate to read cached past states into the current computation, and the paper does not discuss it. How does TransMem relate to it? References for the works named above: Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting Language Models with Long-Term Memory. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). DOI 10.52202/075280-3259.
Reviewer_aWYkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces TransMem, which is a lightweight adapter that modulates hidden states at inference time to compute memory representations. Experiments are conducted on several long-context reasoning benchmarks.

Critical strengths

The proposed loss function is to my knowledge novel. The student model has access to the necessary context and distractor, and it is trained to imitate the self-teacher who only has access to the necessary context but not distractors. This is similar to the well-known context distillation loss, but here the student receives more context tokens, not less. Ablation seems to suggest that this is indeed more effective than just SFT-ing the model to predict the answer from the question and context with distractors. This loss has the potential to be a more effective way to teach a model how to ignore distractors.

Critical weaknesses

A crucial missing baseline is fine-tuning (either full or with LoRA) with the same loss. KL regularization can be used to prevent overfitting if it is a concern. I don't see why this should underperform the proposed method. This solution is simpler, does not require a complicated design of a separate adapter, and is more scalable, especially when the fine-tuning dataset is large and diverse (which it will be, if we are going to apply this loss to actually fine-tune a production model). With fine-tuning, during inference, we do not need to store a separate adapter weight, whereas the proposed method does. Fine-tuning cost can be a little more expensive (it would be worth it to explicitly quantifying), but it is amortized over the model's lifetime after deployment.

The title is misleading in my opinion--it suggests that it is similar to those context-distillation works where you typically remove necessary context from the student and a memory adapter supplies that context. Here, the student has all the necessary information, and the adapter helps with forgetting.

Questions for authors

Why is this better than fine-tuning the model with the same loss?

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive feedback. We have revised the manuscript to address the points raised in the reviews.

  • Section 2 and Appendix B.5. We compare LongMem’s retrieval of cached attention key–value pairs with TransMem’s reuse of sparse segment-final hidden states from the current context’s prefill. Both methods freeze the backbone (e8G5).
  • Section 4.2 and Table 2. We report the mean and sample standard deviation for LoCoMo, HotpotQA, and MemoryAgentBench across 20 evaluation seeds using a fixed Qwen3-4B TransMem checkpoint. Appendix B.3 describes the evaluation protocol (e8G5, 1vyL).
  • Section 4.3 and Table 10. We compare TransMem with full fine-tuning and parameter-matched last-four-layer LoRA under the same ECSD objective. We also report how performance changes when historical states are replaced with random noise at inference (aWYk, e8G5, 1vyL).
  • Table 17-19. We provide the full fine-tuning results and LoRA configuration, along with diagnostics for answer-only training with real history restored at inference, oracle evidence selection, and gold-evidence masking on MemoryAgentBench (e8G5, 1vyL).
  • Table 11 and Table 20. We clarify which HotpotQA training data were used and document that the parametric baselines were evaluated using their original released checkpoints (e8G5, 1vyL).
  • Appendix B.4 and Tables 21–22. We report the added trainable parameters for each backbone and distinguish the frozen backbone’s end-to-end inference cost from the incremental overhead of each memory method at a 10K-token context length (e8G5).

We reply to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.