acceptodds
Under review as a conference paper at ICLR 2027

DPEB: Evaluating Persona Evolution in Multi-Session Dyadic Interaction

Abstract

Long-horizon conversational agents are increasingly expected to sustain a coherent persona while adapting to the relationship they build with a single returning user. The decisive question is temporal and dyadic: does the persona change when, and only when, the user's situation warrants it? Research to date has treated persona as a static target to be steered or a property to be monitored, leaving this question unmeasured. We introduce DPEB, a controlled benchmark for dyadic persona evolution: 1,200 fully reproducible scenarios, balanced by event type at the first evolution point, each pairing an agent persona with one user's relationship arc and labeling objectively the sessions in which a Big Five dimension should move and those in which it should not. Personas are carried by activation steering, measured by projecting the residual stream onto contrastive persona vectors, and scored on three axes (evolution responsiveness, stability, direction accuracy) against an oracle that updates at exactly the labeled sessions. Across the models evaluated DPEB yields a reframing result: static regimes score zero by construction and history-conditioned prompt-update baselines barely move, whereas a re-implemented Persona-Flow-style baseline moves readily but drifts, and holding the update rule fixed while varying only the evidence detector changes the outcome far more; measured on the evaluated scenarios, the two LLM-based gates reach 97-99% event recall yet false-trigger on 36-40% of ordinary (non-evolution) turns, while a prototype detector reaches 96% event recall at a 1.7% false-trigger rate. A lightweight detector trained on sentence embeddings combines both advantages (0.90 event-type accuracy at a 2.3% false-trigger rate) and, with the update rule held fixed, improves all three axes significantly over both deployed gates on two backbones, reaching 93-98% of the oracle's evolution responsiveness. We formalize this as a hierarchical view separating a slow, evidence-licensed trait trajectory from fast within-session adaptation, and we release the scenarios, the frozen user side, the measurement code and the detector diagnostics.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, persona control, activation steering, long-horizon interaction, evaluation benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MfK4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DPEB, a benchmark for evaluating whether agents can adapt their personas at appropriate times during long-term, multi-session interactions while avoiding unnecessary changes. DPEB consists of 1,200 synthetic scenarios, with eight predefined event types mapped to changes in Big Five personality traits. Three metrics (M1/M2/M3) measure evolution responsiveness, stability, and direction accuracy, respectively. Experiments show that detector quality has a greater impact on performance than update scheduling or thresholds. The paper also proposes HPD, a hierarchical personality framework, but it has not been implemented or evaluated.

Critical strengths

The paper addresses when an agent's persona should change and when it should remain stable, going beyond conventional evaluations of static persona consistency.

The benchmark uses fixed user inputs and deterministic labels, enabling controlled comparisons. The three metrics also help distinguish different failure modes of persona updating.

The detector experiments are informative. In particular, the high false-trigger rates of LLM-based gates and the improvements achieved by the trained detector are valuable findings.

Critical weaknesses
  1. The definition of Persona Evolution and the validity of the ground-truth labels are questionable.

Although the paper defines persona as a disposition that remains recognizable across situations and over time, it does not clearly distinguish persona evolution from ordinary task adaptation, user preference memory, or interaction strategy adjustments.

For example, in Appendix A.13, a user's request to try new activities such as climbing or parkour is labeled as requiring an increase in the agent's Openness. However, the agent could satisfy this request and remember the user's preferences without changing its original persona. Similarly, it is unclear why increased stress on the user's side should lead to higher Neuroticism in the agent.

The cited psychological literature on personality changes following life events primarily concerns changes in the personality of individuals experiencing those events. It does not directly justify why events experienced by a user should lead to corresponding personality changes in an agent.

Thus, while the labels are generated deterministically, the validity of the underlying mappings remains insufficiently established. This raises a fundamental question about whether DPEB measures genuine persona evolution or merely compliance with predefined update rules. The authors should clarify the operational boundaries of persona and provide stronger justification for the event-to-trait mappings, potentially through independent human annotation.

  1. Insufficient behavioral validation of dynamic persona changes and a missing baseline to rule out conventional personalization.

Section 4 validates correlations between persona readouts and actual responses using LLM judges, but this validation focuses on static personality levels. M1/M2/M3 are still primarily based on hidden-state projections, which do not sufficiently establish that the measured personality changes persist and manifest in subsequent behavior.

Moreover, the existing baselines mainly compare different persona update policies. An agent that maintains a fixed persona while correctly remembering user preferences and adapting its responses might perform equally well in long-term interactions, yet it would receive zero scores on M1/M3.

I suggest adding cross-session and cross-context behavioral evaluations to verify whether persona updates lead to persistent changes in personality-related behavior. A simple Memory/Preference-only baseline would also help distinguish genuine persona evolution from conventional user adaptation.

  1. The distinction from prior work and the scope of the conclusions need further clarification.

Previous studies, including PERSONA/Persona-Evolve, Do AI Personas Grow?, and PHASE-Tree, have already explored dynamic personality control, event-driven personality changes, and long-term character-state evolution. DPEB's main additional contribution appears to be explicitly labeling when persona updates should or should not occur. The paper could better clarify the unique value of this evaluation design relative to existing benchmarks.

Additionally, the comparison with Persona-Flow uses a reimplementation rather than a full reproduction of the original method. The main experiments also rely on synthetic scenarios with a fixed event taxonomy. Therefore, the conclusion that detector quality is the primary bottleneck is better supported within this controlled setting than for general long-term persona evolution.

Questions for authors

How do the authors distinguish long-term persona changes from task adaptation, user preference memory, or short-term interaction strategy adjustments? What independent evidence supports mappings such as encourage_novelty → Openness+, particularly the assumption that the agent's own personality should change?

Can the authors provide cross-session or cross-context behavioral evidence that persona updates lead to persistent behavioral changes? Could a fixed-persona agent that only updates user preferences achieve similar outcomes?

What is the key contribution of DPEB compared with Persona-Evolve, BFI-Adapt, and LongEvoRoleBench? How do the authors characterize the applicability of their conclusions beyond the predefined event taxonomy?

Reviewer_eujoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies persona evolution in long-term conversations with three main contributions. First, the authors introduce a benchmark DPEP to indicate when a Big Five personality trait should change or remain stable, regarding three metrics: evolution responsiveness, stability drift, and direction accuracy. Second, the authors use Activation Steering to represent and control personality states. Third, the authors propose Hierarchical Personality Dynamics (HPD) framework without validation. Overall, this manuscript has several weaknesses worthy to be addressed in the revision version, such as lack of solid support for event-to-trait mapping, three-metric limitations, synthetic scenarios and fixed user trajectories, etc.

Critical strengths

(1) The research problem is novel.

Different from existing studies mainly focusing on how to make models express specific personality traits and how to maintain persona consistency, this paper raises another question: When should an agent change its personality during long-term interactions, and when should it remain stable? This shift from static persona consistency to dynamic adaptation help study the properties of long-term agents.

(2) The three metrics differentiate failure modes.

The paper introduces three metrics to measure evolution responsiveness, stability drift, and direction accuracy, respectively, by which agents’ different behaviors are identified, such as never update, frequent change, and predefined events-specific update. This multi-dimensional evaluation is reasonable.

Critical weaknesses

(1) The predefined event-to-trait mappings lack solid validation.

The authors map eight user events to specific Big Five personality dimensions and change directions. They describe these mappings as objective labels. However, these labels are objective mainly because they follow fixed rules, not because their psychological validity. These mappings are not always reasonable. For example, when a user experiences significant stress, an appropriate agent may become calmer and provide more stable support. It does not necessarily need to increase its neuroticism. Similarly, when a user asks for less conversation, the agent may temporarily respond in a shorter and quieter way. However, this does not mean that its long-term extraversion trait should decrease. In addition, the eight events cover only some directions of personality change. For example, there are no negative evolution events for agreeableness, conscientiousness, or openness. This limits the evaluation of the full Big Five personality space. Without rational justification for these mappings, a high DPEB score might shows that a model follows the predefined rules, but not necessarily show that the model can perform reasonable persona evolution.

(2) The three metrics have limitations to measure the correctness of persona.

Although these metrics capture some aspects of persona, they do not sufficiently penalize unwanted changes in other personality dimensions. Consider a strategy that changes all five personality dimensions whenever an evolution event occurs. The target dimension changes in the correct direction, but the other four dimensions may also change significantly. During stable periods, the strategy makes no updates. Such a strategy could achieve high M1 and M3 scores while keeping M2 close to zero. However, this is not precise or controlled persona evolution. This happens because M1 and M3 only evaluate the target dimension. Meanwhile, M2 does not penalize unwanted changes in other dimensions during labeled evolution sessions. Therefore, the current metrics do not sufficiently measure whether personality changes are limited to the intended dimensions.

(3) Synthetic scenarios and fixed user trajectories are insufficient to demonstrate effectiveness in long-term dyadic interactions.

DPEB contains 1,200 multi-session scenarios and focuses on dyadic interaction. However, all user utterances are generated in advance and remain fixed. Different agent responses do not affect the user's future behavior or the development of the relationship. As a result, the experiments are closer to controlled dialogue replay than real dynamic dyadic interaction. Although fixed inputs ensure fair comparisons between methods, they also exclude important aspects of long-term interaction. These include user feedback, repeated changes in relationships, misunderstandings and corrections across sessions, and the influence of agent behavior on future interactions. In addition, the main experiments only evaluate the first five sessions of each scenario. Approximately 10% of the evolution events fall outside. This is not sufficient to fully represent longer-term interactions.

(4) HPD’ s connection to the main contribution remains weak.

Despise HPD is not implemented, the main experiments show that update timing and accumulation thresholds have relatively small effects on performance, while detector quality has a much greater impact. The authors do not sufficiently explain why a more complex hierarchical personality dynamics framework is needed based on these findings. They also do not demonstrate what additional benefits HPD could provide over simpler existing mechanisms.

(5) Statistical comparisons remain insufficient.

Although the paper reports paired t-tests and Bootstrap confidence intervals, most configurations were evaluated only once, as well as multi-comparison corrections across the many statistical tests are not performed. In particular, M2 is based on an average of only 1.34 measurable transitions per scenario. Therefore, small differences in M2 should be interpreted with caution.

(6) Syntax errors.

The Abstract on page 1 ends with the incomplete phrase "the two LLM-based gates reach 97-99". The sentence is unfinished and is missing the percentage sign, the metric, and the remaining conclusion. There is also an issue with the confidence interval reference in Figure 5. Figure 5 presents results for n=100, but its caption refers to Table 9, which reports Bootstrap confidence intervals for n=60. In addition, some references in Appendix B are missing publication years.

Questions for authors

(1) Scientific justification for the event labels

Why should an increase in user stress lead to an increase in the agent's neuroticism? Would it not be more appropriate for the agent to remain calm and provide stable support? Have the mappings between the eight events and Big Five personality dimensions been independently evaluated by human annotators or psychology experts?

(2) Independence of persona measurement.

Persona control and evaluation both rely on the same or closely related activation directions. Can the authors provide independent behavioral evidence showing that changes in activation projections correspond to actual changes in personality expression, rather than simply reflecting the direct effects of activation steering? Have the authors considered conducting human blind evaluations on complete multi-session interaction trajectories?

(3) Limitations of the evaluation metrics.

Suppose a strategy changes all five personality dimensions during a labeled evolution event. The target dimension changes in the correct direction, while the other dimensions also change significantly. The strategy makes no changes during stable periods. Could this strategy still achieve high M1 and M3 scores while keeping M2 close to zero? How do the authors plan to penalize unwanted changes in non-target personality dimensions during evolution events?

(4) Synthetic data and detector generalization.

The trained detector achieves performance close to the Oracle. Could this result mainly come from the regular and predictable event expressions in the synthetic data? Can the authors provide additional results on data generated by different models, real multi-session conversations, or challenging negative examples with similar wording but different meanings? It would also be useful to evaluate both event detection and complete persona evolution trajectories on these datasets.

(5) Main conclusions and the role of HPD

The authors argue that detector quality is the main bottleneck in persona evolution. However, the results show that the benefits of better detectors strongly depend on the model backbone. For some models, changing the update timing can produce comparable or even larger improvements. Can the authors clarify under what conditions detector quality becomes the main bottleneck? Also, what unique benefits does the proposed HPD framework offer that cannot be achieved by simpler methods, especially since HPD has not yet been experimentally validated?

Reviewer_QTUkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DPEB introduces 1,200 synthetic multi-session scenarios for evaluating when an agent's Big Five persona should change. Event detectors trigger bounded activation-steering updates, scored for responsiveness, drift and direction. Experiments across four main backbones, plus a smaller-model check, suggest detector quality matters more than update timing. The proposed HPD architecture is not evaluated.

Critical strengths

1.Frozen user turns enable paired comparisons, and detector swaps keep the steering mechanism and update rule fixed. Testing several backbones gives the finding about detector quality more support than a single-model demonstration.

2.The diagnostics expose high false-trigger rates in the LLM gates and test a simple trained detector as an alternative. Full-arc checks and reported trade-offs between responsiveness and drift add useful evidence about the evaluated setup.

Critical weaknesses

1.The event-to-trait labels lack independent validation. Table 13 assumes, for example, that user stress should increase assistant neuroticism. Expert or user annotations are needed to justify lasting personality changes and distinguish them from appropriate response adaptation.

2.The readout uses the steering directions at the steered layer, so direct injection may inflate the scores. Static correlations with two related LLM judges do not validate trajectories. Independent behavioral ratings and injection controls are needed; M1 also rewards magnitude without specifying an appropriate amount of change.

3.Generation and detection share one synthetic taxonomy, with no test of transfer to unfamiliar templates or independently labeled conversations. Five-session evaluations and short full-arc checks provide limited evidence of long-term evolution. Add longer interactions with ambiguous or conflicting events.

4.EGU receives the taxonomy, while the Persona-Flow-style baseline does not; the original systems' tuning and infrastructure are also omitted. Comparisons with equal information and inference budgets, checked against the original implementations, would better establish what the reported gains mean.

5.Detector false-trigger rates use different negatives: 617 stable turns for the trained detector versus 394 for the others (Tables 6 and 8). Rerun them on identical utterances. Single-run comparisons also need replication and correction for multiple testing.

Questions for authors
  1. What independent evidence supports the timing and direction of the target trait changes, especially user stress increasing assistant neuroticism?

  2. Do policy rankings persist under blinded human trajectory ratings and an independent readout that controls for direct steering-vector injection?

  3. How does the trained detector perform on unseen templates and longer conversations with ambiguous events and independently annotated targets?

  4. How do prior-method baselines perform with equal taxonomy access and inference budgets, using or validating against the original implementations?

  5. Why do Tables 6 and 8 use different negatives? Do the advantages survive a shared test set, repeated runs and corrected significance tests?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.