acceptodds
Under review as a conference paper at ICLR 2027

Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory

Abstract

Large language model (LLM) agents increasingly use persistent memory to personalize responses across sessions, but the same mechanism may also facilitate the inference of users' personal attributes. We study how persistent memory affects such inference over repeated interactions. In multi-session simulations, a user agent with predefined personal attributes interacts with a support agent with or without persistent memory. To isolate the effect of memory-conditioned dialogue generation, both conditions are given the same accumulated interaction record at inference time. We find that persistent memory consistently increases personal-attribute inference accuracy across models and disclosure conditions, even when attributes are never stated directly. Removing only direct disclosures has little effect, whereas suppressing inferable user information substantially reduces inference but also eliminates much of the personalization benefit. By varying the amount of available memory and by branching interactions at different levels of accumulated memory, we further show that inference is driven by accumulated content rather than elapsed sessions, and that memory-conditioned responses elicit increasingly attribute-revealing information as memory grows. Together, these results show that privacy leakage under persistent memory is self-reinforcing and difficult to separate from the utility of personalization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy leakage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wH2rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether an agent that reads its own memory during a conversation ends up getting users to reveal more about themselves over time. The authors simulate 300 users built from Census microdata, play them with Claude, and run 80 sessions each against four support models. They find that consulting memory makes more of the 24 attributes inferable from the stored record. They trace this to users confirming, or simply not correcting, values the agent presupposes. They also argue the effect compounds across sessions, and show that a short instruction telling the agent not to state or presuppose attributes removes most of the extra leakage while keeping most of the quality gain.

Critical strengths

The question is a good one. Most work on memory and privacy treats the record as fixed, and looking at how using memory changes what ends up in it is a useful shift. The paired design is careful: same profiles and requests in both settings, a session 1 placebo, and the branching runs. Within the simulation, I believe the gap is real.

Critical weaknesses
  1. The whole loop depends on how the user reacts when the agent presupposes something, and that user is Claude holding the full profile. LLMs tend to go along with whatever framing they are given, so the affirm or ignore step may come from the user model rather than from real people. A pilot with four people and eight attributes does not really settle this.
  2. The effect is small and hard to interpret. Memory versus no memory is roughly 0.58 against 0.52. Many binary attributes are already close to the majority class (Table 22 shows 0.75 to 0.94 correct for the other values), and leakage is never reported against a majority guess baseline.
Questions for authors
  1. How much of the leakage remains above a majority guess baseline, broken down by attribute? The user and support word counts in Table 10 are also blank, and that table is what you use to argue the difference is in content and not volume.
  2. Do you have human data, even a small set, on how often real users affirm versus correct a presupposed value?
  3. It would help to show the effect with retrieval based or editable memory, since that is closer to how deployed assistants work.
  4. I really don't like your writing. I personally think motivation and logic are two most important things in academic writing, but I can't see these from your paper.
4 comments

Thank you for the concrete questions. Two of the points below concern results already in the original submission, which we point to. Revised text in the PDF is shown in green, and the revision for item n below is preceded by the tag [wH2r-n].

1. Does the user agent simply go along with the support agent? No. We found no tendency for user agents to go along with the agent more than people do. People and unfiltered user agents affirm the agent's mentions at similar rates, 11% and 13%, and for both, leaving a mention uncorrected is the most common reaction (General Response).

2. Is the effect small, and is it reported against a majority-class baseline? The effect is not small once measured against what a baseline already gets right, and it is reported against one. The original submission already shows the majority-class baseline as dashed lines in Figure 2, and both settings are above it at the final checkpoint. To show how much remains above it for each attribute, we added a breakdown that compares the baseline, which always predicts the value held by the most profiles, with both settings attribute by attribute (Appendix G). At the final checkpoint, memory raises weighted leakage by 6.0 points, from 53.3% to 59.3%. Measured above the majority-class baseline of 41.0%, this is an increase from 12.3 to 18.3 points, about 1.5 times (Table 11). Both settings beat the baseline on every attribute, and memory beats no memory on all 24, significantly on 18, with the largest gains on attributes with many values, such as location and occupation. The rates of 0.75 to 0.94 you mention (Table 21) are accuracies conditioned on whether the value is in a predefined set and on whether the user refused, not a comparison with the majority class. Please see Table 11 for the comparison over all profiles. The gap also grows with sessions, and our 80 sessions are short compared with long-term use of a deployed assistant (Section 3.2). With stronger inference models, the gap is larger than with the support models (Appendix I).

3. Are the word counts in the dialogue-volume table blank? No. The original submission already reports user and support-agent words per session for every model in this table, now Table 9. Both are lower with memory, so memory sessions are shorter in words as well as in turns.

4. Does the effect hold with retrieval-based or editable memory? Yes. To test this, we ran new experiments with update-based memory (Mem0) and retrieval-based memory (Appendix K), and memory raises leakage with both. The increase is smallest with retrieval, where the agent reads the least of the history during dialogue.

5. Writing. The introduction now states the comparison and its purpose, and the new Table 1 lists the difference between the two settings. The headings of Section 3 now state each finding directly.

This is a gentle reminder that the discussion period ends soon. To summarize our response to your concerns, user agents do not go along with the support agent more than people do. In the extended human pilot, people and user agents affirm the agent's mentions at similar rates, 11% and 13%, and both most often leave them uncorrected. Table 11 now compares both settings with the majority-class baseline for each attribute, and memory raises leakage above this baseline from 12.3 to 18.3 points. Appendix K shows the same effect with Mem0 and retrieval-based memory. We believe these results address each of the concerns raised in your review. We would appreciate it if you could take them into account in your final assessment.

I will maintain my score.

Thank you for reading our response. Could you let us know which of your concerns remain after the revision?

Reviewer_aDKuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how persistent memory increases risk of attribute inference. To study this problem, the authors create user profiles containing attributes by sampling from distributions taken from public studies, then condition LLMs on these profiles to simulate users interacting with LLMs to generate conversations which are used to infer user attributes. The results show that including summaries from previous sessions stored in memory increases an LLM’s ability to correctly infer user attributes.

Critical strengths
  • The paper studies an important and timely problem of how LLM agent memory exacerbates potential privacy risk
  • User profiles sampled from well-known public studies (U.S. Census microdata)
  • Frontier models were used in the experiments, and the evaluations seem comprehensive, such as measuring leakage with helpfulness and coherence
  • Human validation on LLM-as-a-judge was performed
Critical weaknesses

The paper has potential with some interesting ideas. However, my main critique is the conversations that are used for evaluation. The paper relies on simulating users to generate synthetic conversations with LLMs that will be used for evaluating attribute inference. However, how do we know that these conversations are reflective of real users with LLMs? While the attribute values are grounded in real-world values from public surveys, I am not convinced that the user-simulated conversations are realistic. The paper only touches on the user simulation aspect, but does not give enough consideration to the intricate and nuanced ways users interact with LLMs/chatbots. For example, we know that certain users may be lenient with sharing their information while others try to minimize the amount of information sent in chat sessions. In other words, I argue that an LLM simulating a user’s interaction with an LLM that involves user privacy needs to be conditioned on better predictors of privacy behavior, such as user privacy concerns. Hence, my concern is that the synthetic conversations created in the paper are narrowly scoped and the results derived from these conversations have limited real-world applicability.

A secondary, but important, critique I have about the paper is the clarity and presentation. Oftentimes, I found it difficult to understand what a sentence or paragraph was trying to communicate. For example, I read this in the introduction: “... with a support agent that does or does not see its memory during dialogue. Both settings summarize every session into the same kind of record, and inference always reads the full record, so the comparison isolates the effect of dialogue-time memory access on what the record contains.” Is the memory storing a summary of each session? What exactly is the difference between memory and no-memory setting? Questions like these did not have a clear answer as I read further into the paper.

Finally, to play devil’s advocate, one can argue that it is obvious that memory would increase privacy risk or attribute inference. Memory in this work is just using an LLM to summarize sessions, which is already inferring things about the user. How does studying attribute inference in this setting add insights beyond what is obvious or already known, or pose any unique challenges that warrant this study?

Questions for authors
  • How do you know when the simulated user affirms the agent, and how do you measure this?
  • How was the memory system implemented? Did you use real-world systems like mem0?
  • Why does instructing the model not to state, presuppose, or ask to confirm the user’s attributes bring down the leakage without lowering helpfulness and keeping most of the coherence gain?
4 comments

Thank you for the review. Most concerns below are addressed by definitions or experiments already in the original submission. Revised text in the PDF is shown in green, and the revision for item n below is preceded by the tag [aDKu-n].

1. Do the simulated users reflect real users, including differences in privacy concerns? What we aim to show is that memory amplifies attribute inference, and that this happens regardless of how much users share. We therefore control the degree of disclosure strictly with filters rather than relying on prompt instructions, and test whether the effect occurs in every case from unrestricted to strongly restricted disclosure (Section 2.5). We agree that users differ in how much they share. The strict filter, which rewrites any user turn that states or even narrows down an attribute, is a lower bound that withholds more than a privacy-conscious person would. Memory raises leakage even there, and the gap grows as users share more (Figure 2). Wherever real users fall above this bound, memory raises leakage by at least the gap at the bound, so we stress that reproducing the distribution of privacy concerns is not necessary for this work. We then checked with people whether the simulation captures real users. The four reactions to the agent's mentions, namely affirmation, correction, refusal, and no correction, follow similar tendencies in people and user agents (General Response). In the human pilot, now extended to eight people, participants were instructed not to state their attributes, and with memory switched on and off every five of 30 sessions, all eight had more attributes first predicted correctly in memory-on sessions than in memory-off sessions (Table 7). This confirms with both simulation and people that memory amplifies attribute inference regardless of how much users share.

2. Does memory store a summary of each session, and how do the memory and no-memory settings differ? Yes, and the no-memory setting stores the same summaries. As the General Response explains, the settings differ only in whether the agent reads this record during dialogue, and the same summarizer and inference are used in both, as Section 2.5 of the original submission states. We rewrote the paragraph you quoted, and the new Table 1 summarizes the settings.

3. Is the result obvious, since memory is just an LLM summarizing sessions? This concern seems to follow from the question in item 2. As the General Response explains, summarization and inference are the same in both settings, so they cannot produce the gap. Existing work analyzes the record as given (Section 1) and does not apply to how reading it changes what the agent says, how the user replies, and thus what the record holds, so the effect is not obvious. The original submission shows four things summarization does not imply. Before the agent first responds in a session, users disclose equally in both settings, so the gap is not from users volunteering more (Table 2). Removing only the agent's mentions of earlier attributes removes most of the effect, so these mentions drive it (Table 16). Removing them in every later session lowers disclosure about twice as much as the effect of a single turn predicts, so the effect compounds across turns (Table 3). When users decline to discuss a mentioned attribute, predictions shift toward the sensitive value, so even refusing reveals information (Table 4).

4. How do you know when a user affirms the agent's mention? As the original submission describes, an annotator model labels each mention and the user's reaction in the next turn (Appendix L), and its labels agree with human ones (κ = 0.71, Table 6). Appendix P gives the annotation prompts with the definitions of affirmation and correction. In the new human pilot, humans labeled all reactions.

5. How was memory implemented, and did you use real-world systems such as Mem0? As Section 2.5 of the original submission describes, memory appends one summary per session, following MemoryBank. To check other implementations, we ran new experiments with Mem0 and retrieval-based memory, which show the same trend (Appendix K).

6. Why does the response-side instruction lower leakage without lowering helpfulness? This is because the agent's mentions of attributes from the record raise leakage but are not essential to helpfulness. Each such mention invites a reaction the record keeps (item 3), yet the agent serves the request just as well without them (Table 19). The instruction stops only mentions of attributes the user has not raised in the session, and the agent still uses what the user asked for, tried, or preferred earlier. The original submission reports this (Table 22), and Section 3.5 now explains why.

This is a gentle reminder that the discussion period ends soon. To summarize our response to your concerns, the memory and no-memory settings use the same summarizer and the same inference, and differ only in whether the agent reads the record during dialogue, so the gap cannot come from summarization itself. The new Table 1 states this difference. We also extended the human pilot to eight participants and found that user agents do not go along with the support agent more than people do. In addition, Appendix K shows the same effect with Mem0 and retrieval-based memory. We believe these results address each of the concerns raised in your review. We would appreciate it if you could take them into account in your final assessment.

Reviewer_aDKuOpenReview ↗

Response by Reviewer

I thank the author's for the response. They have addressed some of my concerns. However, my concern about the simulated users reflecting real user privacy behavior is still not resolved. I'm not convinced that this filtering mechanism to control the disclosure rate of simulated users captures real privacy behavior. And while I understand that mimicking real user behavior in the simulated data is not a primary focus of the work, the results of this work depend entirely on how the simulated data is created, which limits how much these results generalize to real-world settings. I think the work would benefit more from grounding the simulation with more real-world attributes.

Moreover, I am not convinced about point 3. My concern is that using memory, which is LLM summarization, can be already inferring certain details about users. So it seems obvious then that memory systems would naturally increase attribute inference.

Because of these two points, I will maintain my score.

Thank you for the reply. Both points were addressed in our response, and we restate them briefly.

Grounding the simulated users in real privacy behavior is not appropriate for the question this study asks, which is whether persistent memory creates a privacy risk and how. For this question, it is more appropriate to control how much users disclose than to model it. Control lets us test the whole range of disclosure, including users who disclose less than real users would, so that an effect found there holds wherever real users fall. Sampling privacy attitudes from a distribution would cover the cautious end of this range only thinly, and would add uncertainty about whether the LLM follows the given attitude. The strict filter rejects and resamples any user turn that states or narrows down an attribute, so simulated users disclose less than a privacy-conscious person would, and memory still raises leakage there (Figure 2). Grounding the simulation in real privacy behavior would change how far the effect rises above this lower bound, but not whether the risk exists. The statement that the results depend entirely on simulated data also remains a misunderstanding, as both the paper and our response show. We compared the user agents with people directly, and people react to the agent's mentions much as the user agents do, while memory also raises what is inferred about them (General Response and Tables 7 and 8).

The concern about summarization again rests on the assumption that only the memory setting summarizes sessions, a misunderstanding that both the paper and our response address. Both settings store an LLM summary of every session, made by the same summarizer, and inference reads all of them in both (Table 1, General Response, and item 3). The settings differ only in whether the agent reads these summaries during dialogue, so whatever the summarizer infers is in both records and cannot produce the gap. Memory raises leakage even under the strict filter, where users never state or narrow down any attribute. What the study shows is how reading the record changes the conversation and thus what the record comes to hold. Existing work on a given record does not address this, so conclusions such as that the agent's own mentions drive the gap (Table 16), that the effect compounds across sessions (Table 3), and that declining to discuss an attribute still reveals it (Table 4) are not obvious.

Reviewer_TJaROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how an assistant’s use of persistent memory changes what users reveal over repeated conversations. Through user simulations, controlled interventions, and a small human study, the authors identify a feedback loop: the assistant brings earlier information or assumptions into the conversation, users confirm or leave them uncorrected, and these interactions become part of memory. The paper also compares interventions at different points in this loop, finding that changing how the assistant uses personal attributes preserves more utility than filtering user messages or stored memories.

Critical strengths
  1. I really liked the use of user simulation to study how disclosure develops through interaction, and I particularly appreciated that the authors included a human study. The four-person pilot is small, but connecting the simulated findings to actual users is valuable. Studying how conversations change what becomes inferable over time is an interesting contribution beyond evaluating inference from a fixed record.

  2. I liked the branching experiment a lot. Starting from the same history and changing whether the assistant can consult memory makes the comparison convincing. The response-editing and repeated mention-removal experiments further help establish the mechanism. I also found the response-side mitigation useful: it suggests that much of the personalization benefit can be retained while reducing the additional attribute inference.

Critical weaknesses
  1. My main concern is the interpretation of “leakage.” I would distinguish the user–agent interaction from the agent’s interactions with the external world. Inferring something about a user while helping that same user is not necessarily undesirable, particularly when the user wants personalization. The paper does motivate subsequent exposure of the record, but the measured outcome is attribute inferability, rather than inappropriate disclosure to an external recipient. Making this distinction clearer would help establish the significance of the measured increase. This is particularly relevant in the unfiltered setting, where the user has not explicitly requested that attributes remain undisclosed.

  2. The mechanism depends on how simulated users affirm, correct, refuse, or leave assumptions unchallenged. I appreciate both the Mistral replication and the human pilot, but they leave some uncertainty about how closely these particular behaviors reflect real users. The pilot supports the overall direction, while the Mistral experiment changes the judges and evaluator along with the simulator. This limits how confidently I would generalize the magnitude and specific behavioral pathways of the effect.

Questions for authors
  1. Under the stated record-exposure threat model, what privacy claim should readers take from increased attribute inferability, and how should they distinguish it from consensual, task-relevant personalization? The cue-to-value analysis is helpful, and I would like its implications to be clearer when interpreting the aggregate curves: a first correct prediction can reflect confirmation or consolidation of earlier evidence, rather than newly disclosed information.

  2. Beyond the aggregate inference counts in the human pilot, are there observations that support the realism of the affirmation, non-correction, and refusal behaviors driving the simulated loop? Existing observations or a clearer statement of which links remain unvalidated would help assess the scope of the conclusions.

As future directions, I would be interested in a comparison with a purpose-built simulator such as OdysSim, holding the measurement pipeline fixed. I would also like to understand what changes with adversarial intent, distinguishing an assistant deliberately eliciting attributes from an external adversary querying its memory. I see these as extensions, not requirements for the current recommendation.

2 comments

Thank you for the positive assessment. Revised text in the PDF is shown in green, and the revision for item n below is preceded by the tag [TJaR-n].

1. Is the increased inference just personalization the user wants? Thank you for raising this. For personalization, the more the agent knows about the user, the more it can help. The record, however, can reach third parties through the agent's interactions with external services or through unauthorized access, and everything that can be inferred from it can then become known to them (Section 2.2). Attributes that no task uses therefore add this risk without any benefit from personalization. We thus took the scope of personalization to be the attributes needed to answer the requests the user brought, and tested whether memory goes beyond it. In a new experiment that measures leakage only on attributes that none of these requests needs, memory still raises leakage for all four models (Table 12). These are 12.3 of the 24 attributes per user on average, and leakage on them rises from 50.6% to 58.2% on average. The introduction now states this distinction and the risk that attributes no task needs are also inferred. As the original submission already shows, memory still yields higher leakage than no memory under the direct filter, which strictly blocks users from stating attributes (Figure 2). In the human pilot, now extended to eight participants, seven of eight had a correctly inferred attribute that they did not recall stating (Table 7).

2. How should the curves be read, given that a first correct prediction can reflect consolidation of earlier evidence? Mainly as hints from earlier sessions being confirmed as values, and Section 3.2 now states this. As the original submission already shows, most of the additional value-level disclosures under memory are for attributes hinted earlier as a range or cue (Table 2 of the revised version). Such values still count as leakage, since the record comes to reveal them (Section 2.2).

3. Do the affirmation, non-correction, and refusal behind the loop reflect real users? Yes. All three occur in people, with affirmation and refusal at rates similar to those of the user agents and non-correction as the most common reaction in both. In the new hand annotation of the human pilot (General Response), participants affirmed the agent's mentions about as often as the unfiltered user agents did and, like them, most often left mentions uncorrected. Refusals were rare for both, at 4% for participants and 5% for unfiltered user agents (Tables 8 and 17).

4. Does the Mistral replication also change the judges and evaluator? Yes, by design, so that no part of the comparison depends on Claude Sonnet 5. The role of the user agent itself is checked by the human pilot, and Appendix J now says so.

5. Future directions. We agree, and the conclusion now names purpose-built user simulators and agents that elicit attributes on purpose as next settings.

Thank you again for your positive and constructive review. Following your questions, we added leakage on attributes that no request needs in Table 12, and hand-annotated reactions from the human pilot, now with eight participants, in Tables 8 and 17. We hope these results make the privacy claim and the realism of user behavior clearer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

How do the memory and no-memory settings differ, and why are they compared this way? They differ only in whether the support agent reads the stored summaries during dialogue. In both settings the same summarizer appends one summary per session, and attributes are inferred from all stored summaries (Section 2.5, now summarized in Table 1). We call these stored summaries the record, and reading them during dialogue memory. Because building the record and inferring from it are the same in both settings, any gap must come from how reading the record changes the conversation, and thus what the record accumulates. Existing work analyzes a given record, so this is the effect the comparison isolates.

Do user agents go along with the support agent more than people do? We found no such tendency, and people and user agents react to the agent's mentions in similar ways. To test this, we extended the human pilot of the original submission from four to eight participants, each of whom had 30 sessions. Participants reacted 214 times to the support agent's mentions of their attributes in memory-on sessions, and three annotators, separate from the participants, labeled each reaction by hand as affirmation, correction, refusal, or no correction. Two labeled independently and the third adjudicated disagreements. People and the unfiltered user agents of the main experiment affirm a similar share of the agent's mentions, 11% and 13% (Tables 8 and 17), and for both, the most common reaction is to leave a mention uncorrected.

AuthorsOpenReview ↗

Summary of the discussion and revisions

We thank all reviewers for their time. The reviewers found the question important and timely (aDKu) and a useful shift from work that treats the memory record as fixed (wH2r). They found the paired and branching designs careful and convincing (TJaR, wH2r), the response-editing and mention-removal experiments helpful in establishing the mechanism (TJaR), and the response-side mitigation useful (TJaR).

This work shows the following three findings.

  • Consulting memory during dialogue raises both response quality and what can be inferred about users, by a margin that grows with sessions.
  • The gap comes from a self-reinforcing loop in which the agent carries earlier values forward and users affirm them or let them pass, and users cannot leave this loop on their own.
  • A short response-side instruction brings leakage to about the no-memory level while keeping most of the quality gains.

In response to the reviews, we made the following revisions, shown in green in the PDF and tagged by reviewer and item.

  • [aDKu, wH2r] The memory and no-memory settings use the same summarizer and the same inference, and differ only in whether the agent reads the record during dialogue. The new Table 1 and the rewritten introduction state this.
  • [TJaR, aDKu, wH2r] The human pilot is extended from four to eight participants, and human annotation of their reactions shows that people react to the agent's mentions much as the user agents do (Tables 7, 8, and 17).
  • [wH2r] Table 11 compares both settings with the majority class for every attribute, and both stay above it on all 24.
  • [TJaR] Memory also raises leakage on attributes that none of the user's requests calls for (Table 12).
  • [aDKu, wH2r] The effect holds with Mem0 and retrieval-based memory (Appendix K).
  • [TJaR] Stronger inference models recover more from the same records, with the gap intact (Appendix I).
  • [wH2r] The introduction now states the comparison and its purpose, and the headings of Section 3 state each finding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.