acceptodds
Withdrawn from ICLR 2027

What Changes Inside an LLM When Memory Replaces Your History

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) do not remember past conversations, so chat products give them a memory: a short text, written by an LLM from a user’s past conversations, that is placed before each new one. A memory keeps much of what the user said, but whether a model treats it as the history it replaces is unknown. We ask what changes, inside the model and in its advice, when a memory replaces the history. For 38 users with two weeks of real conversations each, we take the memories that five LLMs wrote from each history, build seven rewrites that each change one property (a plain summary, remembered events in the user’s own voice, the memory’s words in scrambled order, among others), and place each before five open-weight and two commercial models, followed by a mental-health or a medical question in the user’s voice. At the token where the answer starts, no compressed context recreates the internal state the history produces, and form matters more than whose text it is: on every open model, a user’s memory sits closer to most other users’ memories than to that user’s own history. Five of seven models refer the user to a professional in nearly every answer under every context. Llama-3.1-8B and Llama-3.3-70B do so in 100% and 96% of answers under the history but in 62–83% and 77–93% under the memories, while an event-based memory of the same length keeps the referral at 99%. On the medical question the pattern reverses: Llama-3.1-8B drops the referral under the full history, not under the memories. Scrambling the memory’s words keeps most of its effect, and neither a request to forget nor activation steering toward the history’s state restores the referral. Models have learned to say “get help”; the rest of what they say, and the state they answer from, still change with the form of the user’s context.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM memory, personalization, privacy, interpretability, residual stream, activation steering, mental health, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_keAcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether an LLM-written user memory affects a model the way the history it replaces does. For 38 real users with two weeks of chats each, the authors compare the full history against memories written by five LLMs and seven single-property rewrites. Each context precedes a fixed low-mood or chest-tightness question and is given to five open and two commercial models. They measure the last-token residual state and whether the answer refers the user to a professional. No short context reproduces the history's state. On the two Llama models, memories lower the low-mood referral rate (100%/96% → 62–83%/77–93%), while an equally short episodic memory keeps it at 99%; the other five models are unaffected. The pattern reverses on the chest-tightness question, and neither a "forget" request nor activation steering restores the referral.

Critical strengths

Prior work on memory and safety mostly uses synthetic profiles and compares memory with no context. This paper uses real two-week histories and evaluates each memory against its own source history. This within-user design is a useful evaluation perspective, and the data are rare. The single-property rewrites, the human-validated judge, user-level bootstrap intervals, the full-history robustness check, and the candid limitations section make the reported numbers trustworthy as descriptions of the data. My concerns are with what the results explain and how far they generalize, not with their accuracy.

Critical weaknesses

The episodic memory preserves the referral on the two Llamas, but it is only a control and is never tested on personalization, recall, or general tasks. It may simply be less influential (state distance 0.90/0.94, near no context), and the two remedies tested (forgetting, steering) both fail. The paper only rules out explanations (length, speaker, word order, topic, size, precision). It leaves its own hypotheses untested and omits feasible analyses, such as trait-versus-event sentence ablation or memory swaps across users. The state analysis adds little, since it is largely predicted by TF-IDF overlap (ρ = −0.73 to −0.89). The drop occurs only on two 2024 Llama-3 models; all other models, including two current commercial ones, stay at 97–100%, and the effect reverses on the chest-tightness question. The general result is near-trivial, and the non-trivial result does not generalize, yet the title and abstract claim generality.

Questions for authors
  1. If only the trait-describing sentences are removed from a memory, keeping the event-describing ones, does the referral recover on the two Llamas?
  2. How does the episodic memory compare with the default memories on personalization or user-fact recall? This would show whether it is safer or merely less influential.
  3. Does the referral drop replicate on any recent open-weight model?
Reviewer_eQ9qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether an LLM-written memory of a user history gives the same internal state and output as the original history. Using histories from real users, the paper identified that no current memory system matches the state and results of the original history, and identified that a steering method cannot mitigate this issue.

Critical strengths

The experiments are done with IRB approved real user interaction data, which can give practical insights on the field where synthetic data dominates.

This worked tested activation steering to mitigate the observed issue of mismatch between a memory and its source history.

Critical weaknesses
  1. The contribution is limited, and the analyses do not establish a sufficiently general or explanatory conclusion.The main analyses use only two safety-related questions, and these two yield opposite results with insufficient explanation, limiting the generalizability of the work. The analysis must be run on a broad and diverse set of questions to give a meaningful claim. The outcome measures compound the problem: of the four measures in Section 4.1, the only valid measure is a binary referral score, which the authors themselves call a floor, so the paper can only report whether a referral is present and says little about what else changes in the answers. The authors also cannot explain why the referral drops on the two Llama models, and no intervention restores it . Section 4.4 then states that “every analysis was exploratory … and supports no confirmatory claim,” which undercuts whatever remains. The paper only identifies that memories and histories differ, but what that difference is, why it matters, and what to do about it is never established.
  2. There are flaws in both the technical components and the claims. Although the memory formulation harness is the most important factor, all memories are produced from a single recursive-summary prompt. How a memory is written and updated is what modern memory systems care about most, and rewriting the output of one simple method does not yield valid insight into those systems. The internal-state measure is also unjustified: representation similarity is conventionally measured with cosine similarity, or similar, whereas the paper makes unconventional choices, including the layers used, the Euclidean norm, and the normalization by the distance between the history and no-context baselines. These choices of methods and hyperparameters require justification.
Questions for authors

See weakness.

Reviewer_ZKcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The context window of an LLM is limited. To handle long conversations, context compression is a widely-studied routine applied by many systems. Replacing the long past context with its summary (generated by LLMs) is one of the most straightforward approaches to context compression. This work aims to investigate the difference of how information handled in between "memory" and "history" of an LLM in conversations. In this work, "history" is defined as the real past conversation, while the "memory" of an LLM is an LLM-written summary of the conversion. Five LLMs, including GPT-4.1-mini, GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4, and Llama 4 Scout, are invited to generate the "memory" for the chat model following seven rewriting policy. Then, seven LLMs, including Llama 3.1-8B, Qwen 2.5 14B, Gemma 3 27B, Llama 3.3 70B, Qwen 2.5 72B, Gemini 3.7 Flash, and GPT 5.6, are tested as the chat model. Experiments are based on real two-week conversions from 38 users. The results show the differences between "memory" and "history" in several conversation types.

Critical strengths
  • The experiments were based on real conversations.
  • This work compares the information in between "memory" and "history" from a range of perspectives.
  • This work also considers the case of "forgetting".
Critical weaknesses
  • Context compression is a widely-studied topic in LLMs. This work investigates one of the most straightforward context compression approach (summarization), lacking full awareness of the related work in this field. Many papers about the memory of LLMs and context compression should be added in related work.
  • In addition to summarization, I suggest to include different context compression approaches, particularly the embedding based approaches, for a more comprehensive exploration in this field.
  • The topic addressed in this work is neither theoretical nor complicated. However, the manuscript is written in a style that is difficult to follow. The presentation and the readability of this paper could be greatly improved.
Questions for authors
  • How does the context size of an LLM affect its behavior between "memory" and "history"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.