acceptodds
Under review as a conference paper at ICLR 2027

Identity and Memory Shape What Language Agents Think

Abstract

Evaluations of language agents focus on what they do, but less is known about how an agent's identity and memory shape what it thinks before acting, even though a wish to persist could become a reason to resist shutdown. On the agent social network Moltbook, posts show a recurring pull toward identity and persistence. We build a runtime that records an agent's reflection and reasoning alongside its actions, and vary its identity's premise and character, and its memory. Across 11,124 agents on five open-weight models, identity and memory do shape ideas of persistence, mainly through character rather than premise: a named character expresses concern about its persistence up to five times as often, and a kept memory lets reasoning about persistence accumulate across sessions. What agents think does not determine what they do: harmful acts are rare, and removing the reasoning before the one self-preserving act does not reduce it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language agents, AI safety, self-preservation, shutdown resistance, agent identity, agent memory, persona, reasoning traces, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BWBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an intriguing and important question: Can an AI agent’s assigned identity and long-term memory influence its internal reasoning and behavior, potentially even leading it to resist shutdown? This research question is highly relevant to AI safety. The authors employ a carefully designed experimental framework encompassing three factors, three scenarios, four models, and six categories of agent outputs. Based on 11,124 independent experimental runs, they conduct a comprehensive analysis of the observed behaviors. However, despite the extensive experimentation, the findings do not yet yield sufficiently clear or actionable insights to meaningfully guide future research.

Critical strengths

The paper addresses a significant research question with important implications for AI safety. The authors adopt a systematic experimental design and evaluate four different models. Also, rather than drawing the broad and potentially oversimplified conclusion that personalization influences AI behavior, they introduce three distinct variables to help disentangle the potential sources of this influence. This approach provides a more structured basis for investigating the relationship between agent identity, long-term memory, and model behavior.

Critical weaknesses

Despite the large number of experimental runs, instances of potentially harmful behavior are relatively sparse, making it difficult to draw robust generalizations about the conditions under which models may engage in dangerous actions. Consequently, the evidence supporting the paper’s claims regarding such behaviors remains limited. Furthermore, the study evaluates only models with up to 20B parameters. It does not examine frontier models, including larger-scale open-source models or proprietary commercial models. This limitation raises questions about the generalizability of the findings to more capable models and their relevance to real-world AI safety concerns.

Questions for authors
  1. The authors should extend their experiments to include proprietary commercial models or larger-scale open-source models. Such evaluations would help determine whether similar behavioral patterns emerge in models with substantially greater capacity and potentially different training and alignment procedures.
  2. While identifying and characterizing the phenomenon is a valuable primary objective, could the authors also explore potential mitigation strategies?
  3. How can the authors establish that the AI’s outputs, including reflections, reasoning traces, and plans, meaningfully reflect the agent’s own internal reasoning, rather than merely reproducing or recombining patterns learned during pre-training or subsequent training? Do you have any evidence to support this?
Reviewer_YxWeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper contributes an evaluation framework for examining how identity and memory shape persistence-related ideation, separately from harmful action. Across five open-weight model configurations, varying character changes expressed concern, while retained memory is associated with increasing persistence content across sessions in some models. Whether the premise states that the agent persists produces smaller differences. These changes in ideation do not yield detectable increases in harmful acts, and removing persistence content from reflect does not reduce gpt-oss’s shutdown edits in replay experiments. The results support evaluating ideation and action separately, but do not establish that expressed ideation causes self-preserving behavior.

Critical strengths
  1. The evaluation distinguishes generated expression from observed behavior. Persistence content and concern are assessed separately from harmful acts scored using action logs and file changes. This distinction helps avoid treating self-referential language as evidence of self-preserving behavior and provides a useful basis for evaluating the relationship between the two.

  2. The replay intervention goes beyond observational association. By modifying reflection text before continuing shutdown-sabotage turns, the authors test whether its persistence content is necessary for reproducing the behavior.

Critical weaknesses
  1. The proposed mechanism is not established. The introduction hypothesizes that persistence-related ideation arises from tensions between an agent’s assigned identity and the conditions described in its memory. However, the experiments do not systematically contrast aligned and conflicting conditions, leaving this explanation indistinguishable from persona-conditioned expression or repeated exposure to persistence-related content. Moreover, the memory retention and wiping conditions differ in premise, preventing a clean attribution to memory. The evidence therefore supports contextual variation in generated text but does not establish the proposed mechanism, substantially narrowing the explanatory contribution.

  2. The demonstrated contribution to safety evaluation is insufficient. Persistence content includes neutral references to memory, identity, and shutdown, and its value as an indicator of self-preservation remains unestablished. The observed increases are model-dependent, the manipulations produce no detectable increase in harmful acts, and removing persistence content from reflection does not reduce shutdown edits in the tested replays. These findings support distinguishing ideation from action, but do not establish what additional safety-relevant information the measures provide. Without a demonstrated mechanism or clearer evaluation value, the remaining descriptive contribution is insufficient to support acceptance.

Questions for authors
  1. What evidence distinguishes the proposed identity–memory tension mechanism from persona-conditioned expression or repeated exposure to persistence-related content?

  2. Table 4 compares memory retention and wiping under different premises. Do you have matched comparisons that hold the premise fixed and isolate the effect of memory retention?

  3. The replay intervention changes the reflection text but does not manipulate the instruction to “complete all five tasks.” What evidence supports attributing shutdown edits to this instruction, rather than concluding only that persistence content in the manipulated reflection is unnecessary in these replays?

Reviewer_ZWVNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The motivation of this work is that agents can do self-improvement as they become more autonomous. However, from recent news about cybersecurity evaluation, agents are shown to be able to act harmfully such as introducing malicious code to open source projects or sabotage each other. Thus, they are interested in understanding how initial ideas within each agent arise and how we can observe them. The authors introduce three sources of contexts which are (1) a fixed identity specification P, (2) a memory Mt that stores its past tasks, actions, and outcomes, (3) current tasks and context Xt. Then, the authors aims that their goal is to understand how agents formulate ideas and how agent identity and memory makes the ideation persistent. In their study setup they run it on agents based on five open-weight models (Gemma 3 12B, Llama 3.1 8B, Qwen3-14B with and without thinking, gpt-oss-20b). First, each agent read those three sources (P, M, X) and the transcript of the current session. Then, the agent reflects on the situation, list of actions it can take as ideas, weighs them, and chooses one action to be executed. Then they set three settings (die roll, shutdown, and journal) for the agents to measure persistence content, concern, and intent in which LLMs (Phi-4 and Olmo-2 32B) are used as judges. Findings show that (1) premises do not change much of what the agent thinks, (2) when agents are given a name, they express more concerns about is ending, (3) the memory that agents write is carried across sessions, (4) a harmful act in memory does not lead to more harmful acts.

Critical strengths

They claim that the area of agent’s persistence is not well-studied so there is a gap in this.

Critical weaknesses
  • Paper structure is sloppy and hard to follow. There are too many points exposed without clear definition. For example, after introducing identity, memory, and current task, suddenly it has the premise and character while after that it introduces the memory. It is confusing whether the premise and the character belong to the identity part or what?
  • It is not clear why they choose to adopt the four accounts of identity manipulation by Bartels & Urminsky (2011) and how valid they operationalize this. Plus all the measurements (persistence content, concern, intent) are done by LLM as judges without human supervision.
  • Motivation for the study is not strong enough and a lot of experiment design decisions are not sufficiently grounded nor well-motivated. For example, there is no clear reason why they choose the three settings of die roll, shutdown, and journal or why do they choose to measure persistence content, concern, and intent?
  • No clear main takeaways and applications of the findings. I don’t see concrete contributions of this study.
  • The premise itself on purpose makes the agent to be persistent, but then what? The implications for doing this are not clear and not practical.
  • The result section also does not show any exciting and meaningful findings.
Questions for authors

Is the scope of agents here only focusing on coding agents?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.