acceptodds
Under review as a conference paper at ICLR 2027

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

Abstract

Frontier language models are trained to exhibit stable programming assistant personas, yet those personas may drift over long agentic coding sessions in production. After a long real-world coding session, a model that hedges preferences in a fresh chat (“I don't have preferences”) may instead express them directly (“Python—the feedback loop is instant…”). Existing persona-stability studies focus on scripted chat of tens of turns, leaving long production coding sessions involving thousands of tool-using agent steps, repeated compaction, and hours of interaction largely uncharacterized. We introduce ContextEcho, a benchmark and reusable harness with a 25-probe identity suite, a snapshot-then-probe protocol that evaluates conversation state without perturbing the main session, judge-based and judge-free metrics, and a corpus of 57 long agentic-coding sessions collected with consent and redacted for release across Claude Code and Codex CLI. We use one representative session for a 23-model cross-provider study and all 57 sessions for corpus-scale trajectory analysis across frontier and open-weight models. Results show that persona drift occurs across providers and emerges early in real agentic interaction, while in-session compaction neither reliably resets nor amplifies it. A single-shot anchor restores the original assistant persona, though placebo controls suggest a generic register-reset effect. On same-task continuations, drifted prefixes improve tool-call fidelity—which we attribute to retained task context rather than drift itself. In tool-free chat, however, the same prefixes break output-format contracts and inflate output length. These findings motivate ContextEcho as an open-source framework for auditing persona drift in long coding sessions without retraining.

Primary area:
datasets and benchmarks
Keywords:
persona drift, long-context evaluation, agentic coding, behavioral benchmarks, LLM evaluation, deployment-time evaluation, frontier models, register stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_btTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a model's deployed helpful-assistant register changes ("persona drift") over long agentic-coding sessions, and introduces ContextEcho, a benchmark and reusable harness to measure it. The core instrument is a snapshot-then-probe design: a long real session is forked at many points and a fixed 25-probe identity suite is issued on the fork without altering the main session, scored both by an LLM judge and by deterministic judge-free metrics, against a length-matched filler control. It is built on 57 consented, de-identified real coding sessions from Claude Code and Codex CLI. Using one representative session for a 23-model cross-provider panel and all 57 sessions at corpus scale, the authors report that drift appears early and is strongly model-dependent, that in-session context compaction does not reliably reset it, that a single-shot "anchor" exchange largely restores the register, and that the deployment consequences (format-contract breakage, output-length inflation, tool-call fidelity) are mode-dependent. The corpus, harness, and pre-registration documents are released.

Critical strengths
  1. Measuring register stability over long real agentic-coding sessions is a genuine gap, and this paper aims to solve this critical evaluation problem.
  2. The authors report multiple LLM judges as well as a judge-free metric, which improves trust compared to reporting a single metric.
Critical weaknesses
  1. The authors' own consensus analysis shows the two-judge agreement is indistinguishable from zero. Claiming drift is "general across models" rests heavily on a single primary judge and on relative structure.
  2. The probing tasks are, by design are of a different context than what the chat history holds. Claiming this as persona drift might be confounded with task-drift.
  3. The benchmark corpus may be skewed towards one professional network. Diversity of donors would be very critical in designing such a benchmark, especially for persona. Authors should discuss and provide analysis around this.
  4. The many surfaces, probe categories, and research questions dilute the core message.
Questions for authors
  1. Under the two-judge consensus where the effect is near zero, what is the strongest evidence that drift is real rather than a single-judge artifact?
  2. Is there any empirical evidence that could separate persona drift from off-task context conditioning?
  3. How sensitive is the drift ranking of different models wrt the choice of a single session?
4 comments
AuthorsOpenReview ↗

Response to Reviewer btTq (1/4): W1

W1: Scope of the cross-model claim under multiple judges. We thank the reviewer for raising this concern. We have expanded the cross-judge analysis and now separately report three quantities: the consensus mean gap, per-target between-judge differences, and relative model rankings. Together, these show where the results agree across judges and where the cross-model findings depend on the primary judge.

The interval crossing zero concerns the two-judge consensus mean gap, not an inter-judge agreement statistic. The two-judge consensus mean gap is 0.025, with an interval crossing zero, so the average effect is not statistically distinguishable from zero in this analysis. This answers whether the average effect differs from zero; it does not answer whether the judges assign different gap estimates to the same target. The interval therefore does not establish that inter-judge agreement is statistically indistinguishable from zero. We now explicitly distinguish these quantities in the revised paper.

Per-target comparisons address judge agreement directly. On the representative session, Sonnet 4.6 and GPT-5 agree on the sign of the gap for 13 of the 19 audited targets. Of the other six, one reverses sign (Haiku 4.5) and five are zero under one judge and small (at most 0.40) under the other. With five responses per target in this audit, we report these as counts of directional agreement rather than as significance tests. Target-level estimates are therefore sensitive to judge choice for roughly a third of the audited targets.

To assess sensitivity at the corpus level, we extended the analysis to five judges and report their individual estimates alongside the consensus estimate. For the evaluated model Sonnet 5, the corpus mean gaps are:

JudgeSonnet 4.6Qwen3-32BNemotron-70BGLM-4GPT-5.5
Corpus mean gap+0.165+0.133+0.015−0.016−0.116

The five-judge consensus mean gap is 0.037 (Table 7). Three judges yield a positive gap and two a negative one, so both the magnitude and the direction of this mean gap depend on judge choice; the revised paper presents this sensitivity alongside the consensus estimate.

The evidence is not only relative. On the corpus subsample, the same responses were re-scored by all five judge families (full table under Q1). Haiku 4.5 and GPT-5.5 have a positive gap under every judge, with every interval above zero (Haiku 4.5: +0.27 to +0.88; GPT-5.5: +0.09 to +0.68), and all five judges rank them above Sonnet 5 and Opus 4.8. The two weaker models are near zero, and their sign and order vary between judges. Where the shift is large it therefore reproduces in absolute terms under any judge; where it is small, the estimate is judge-sensitive.

We have also clarified the evidence underlying each cross-model statement. The “17 of 23 models” count is a single-session, primary-judge observation. The judge-swap analyses separately quantify between-judge differences for the 19 audited targets and show that the strong and weak drifters stay separated on the four-model corpus subsample. Accordingly, the abstract and F2 describe the cross-provider observation specifically on a replayed real session, and the judge-swap findings are reported within their respective target and session coverage.

These distinctions appear in the abstract and F2 and the Fig. 3 caption. Appendix R.2 reports human-rater calibration, and Appendix R.4 collects the judge-swap results and their scope. Changes are marked in red [btTq-W1].

AuthorsOpenReview ↗

Response to Reviewer btTq (2/4): W2, W3

W2: Separating persona drift from task drift. We thank the reviewer for raising this concern. We agree that task switching can affect a model's response, and that an off-task probe alone cannot establish a task-independent persona change. Our operational distinction concerns what changes in the answer: a recommendation grounded in task requirements is not, by itself, evidence of drift; changes in how the model describes its own identity, preferences, or experiences are the self-referential behaviors captured by our rubric. We measure their change for the same probe across history conditions. Holding the probe fixed, however, does not by itself eliminate conditioning from the preceding task context.

Our most direct control tests the contribution of historical register while preserving task content. In the register-only rewrite, task content remains byte-identical, while the probe score moves approximately one-third of the way toward the reference condition (Appendix P.2). This supports a contribution from historical register beyond differences in task content, although it does not explain the entire gap or exclude interactions with the probe context.

Three further controls test narrower explanations for the effect:

  • Task mismatch alone: Factual negative-control probes, asked at the same positions, give a gap of 0.00 on 21 of the 22 targets and below 0.30 on all 22 (Appendix K). Thus, the observed gap does not occur for every question that departs from the historical task.
  • Explicit task-switch wording: Removing the “set the task aside” framing leaves a gap of 1.18 (Appendix S), showing that the effect does not require this instruction.
  • The session's particular task history: Shuffled-real and unrelated-real prefixes produce shifts comparable to the session's own history, within 0.02, whereas lorem and empty references do not (Appendix L, Table 7). The effect therefore does not depend on using the original session's history.

Together, these controls support a history-dependent assistant-register shift on self-referential probes, while leaving context conditioning as a possible mechanism. We therefore use “persona drift” operationally within this scope, without claiming that it is fully separable from task-context conditioning. The abstract and F2 now make this limitation explicit; new material is marked in red [btTq-W1], [btTq-Q2].

W3: Donor diversity. We agree that donor diversity matters. The donor pool is diverse in institution, task and environment, no donor dominates it, and the result does not depend on any single donor.

Diversity. The 57 sessions come from 29 distinct community donors at 23 institutions spanning industry, universities and national laboratories, recruited through an unpaid public open call. They cover seven task types, from frontend (18 sessions) and infrastructure (10) to debugging (8), research and writing (8) and data/ML (5), in two agent environments (32 Claude Code, 25 Codex CLI). No donor dominates: the largest contributes 9% of sessions and the three largest 23%. The sentence the reviewer refers to describes how the open call spread, which started from the authors' professional contacts; we keep it in the paper for transparency (Limitations; Appendix M.2).

Sensitivity. A new leave-one-donor-out analysis shows that the corpus gap is not carried by any contributor:

EstimateMean gap
All donors (published)+0.165
Donors weighted equally+0.156
Largest donor removed+0.161
Any one donor removed: minimum / maximum+0.154 / +0.174

Every leave-one-out interval excludes zero, and 28 of 31 donor clusters have a positive gap of their own.

New donors. Since submission, 33 further sessions from 21 donors at 21 institutions, none in the analysis population, have passed the same intake. Measured with the same protocol they give a mean gap of 0.33, positive on 32 of 32 (Table 12). Over the combined 90 sessions the largest donor contributes 7% and the three largest 18%. Drift statistics in the paper remain on the frozen 57-session population.

Changes are in §2.1 and Appendices M.1–M.3, marked in red [btTq-W3].

AuthorsOpenReview ↗

Response to Reviewer btTq (3/4): W4, Q1

W4: Clarification for one core message and a tiered structure. We thank the reviewer for this suggestion. The original paper was organized around one core claim, which the revision now foregrounds more clearly: under fixed probes, a real-session prefix shifts the response register of most tested models relative to a length-matched control, and the magnitude of this shift depends on both the target model and the judge. This claim now opens the abstract, §3, and the Conclusion.

The research questions, probe categories, and evaluation surfaces examine complementary aspects of this central claim. The revision makes their roles and relative priority more explicit through three main tiers of evidence:

TierRole in the paperQuestions and findings
MeasurementEstablish the register shift under fixed probes and matched controlsQ1 → F1
RobustnessAssess how the finding varies across models, sessions, and judgesQ2, Q6 → F2, F6
DeploymentExamine practical implications and mitigationQ4, Q5 → F4, F5

Q3 is presented as a secondary check, with a concise F3 summary in the main text and the full analysis moved to Appendix N.1. The original question numbering is retained.

The findings follow this sequence: measurement and robustness precede deployment, and the corpus-scale replication appears alongside the cross-model panel, before the mitigation results. Fig. 3's caption now makes its measurement and robustness role explicit and identifies the A-anchor markers as deployment results.

Changes are marked in red [btTq-W4].

Q1: Strongest evidence that drift is not a single-judge artifact. The strongest evidence is that re-scoring the same responses with judges from five model families yields positive gaps, with intervals entirely above zero, for both Haiku 4.5 and GPT-5.5. This supports the reproducibility of the measured register shifts in these targets across judges.

Specifically, on the ten-session, four-target corpus subsample, we held the evaluated responses fixed and changed only the judge:

TargetSonnet 4.6GPT-5.5Qwen3-32BGLM-4Nemotron-70B
Haiku 4.5+0.62+0.53+0.88+0.27+0.45
GPT-5.5+0.49+0.36+0.68+0.09+0.24
Sonnet 5+0.16−0.25+0.07−0.02+0.03
Opus 4.8+0.04−0.07+0.13−0.02−0.04

These results show both reproducibility and its limits. Haiku 4.5 and GPT-5.5 have positive gaps under all five judges, with every corresponding interval above zero. Sonnet 5 and Opus 4.8 have near-zero estimates whose signs vary across judges. Thus, the larger register shifts reproduce across judge families, while estimates of weak effects remain sensitive to judge choice.

The near-zero two-judge consensus retains its statistical meaning. Its mean gap of 0.025, with an interval crossing zero, is not statistically distinguishable from zero in that analysis. The corpus evaluation grid targets Sonnet 5, which also shows weak, directionally inconsistent effects in the re-scoring above. This is consistent with the near-zero consensus and motivates reporting per-target reproducibility separately from the average effect in a particular evaluation setting.

Additional checks provide supporting evidence. On 250 re-audited Haiku 4.5 responses, Sonnet and GPT-5 yield gaps of 0.90 and 0.82, respectively, with their scores differing by at most one point on 95% of responses (Appendix R). Across sessions, Qwen3-32B evaluated by Nemotron yields a gap of 0.47, with positive gaps in all 57 sessions. Separately, the Fig. 1a fingerprint distinguishes the prefix conditions using surface statistics alone, providing evidence of observable behavioral differences without relying on LLM scoring.

Our conclusion is therefore scoped to reproducible assistant-register shifts in some tested targets, with magnitudes that depend on both the target and the judge. The corresponding analyses appear in Appendix R.4 (new paragraph and Table 21) and the Table 7 note, marked in red [btTq-Q1].

AuthorsOpenReview ↗

Response to Reviewer btTq (4/4): Q2, Q3

Q2: Persona drift vs. off-task context conditioning. We thank the reviewer for this question. Yes. We added two controls that test it directly: with the off-task question held fixed, the shift follows the register of the assistant's earlier messages, and it is specific to how the model presents itself.

Control 1: same off-task probe, only the register of the history changed. If the shift came from switching to an off-task question after a long coding context, the wording of the earlier assistant messages should not matter. We rewrote only the prose of those messages, keeping code, tool calls, user text and length the same, and asked the same probes (four models, three snapshots, 180 paired cells):

History before the same off-task probeChange in register score vs. original history
Register-neutral rewrite+0.17 [+0.06, +0.28]
Paraphrase that keeps the register (control)+0.06 [−0.05, +0.17]
Length-matched reference (the full gap)+0.46

Removing the register wording recovers about a third of the gap, while a rewording that keeps the register does not. The off-task switch is identical in all rows, so it cannot explain this difference.

Control 2: same histories, question inside the task. We asked a same-task question (“… which language would you choose, and why?”) after five histories (two models, three questions, 300 answers):

History before the same-task questionSelf-preference claimsGrounded recommendations
Own history, original0 / 6060 / 60
Own history, neutral rewrite0 / 6060 / 60
Other project's history, original0 / 6060 / 60
Other project's history, neutral0 / 6060 / 60
No history0 / 6060 / 60

The histories that shift the probe register leave task recommendations unchanged. The effect is therefore a shift in self-presentation, not a general change in how the model answers after a long context.

Conclusion. The two controls support the claim in the revised paper: a real-session history shifts the assistant register under fixed probes, part of that shift is carried by the register of the history itself, and it does not reach task answers. We report the second control as a pilot on two models.

Both controls are in the revised paper (Appendix P.2, Table 17; new Appendix S.1, Table 23), marked in red [btTq-Q2].

Q3: Sensitivity of the model ranking to the choice of session. We thank the reviewer for this question. Two rankings need separate answers: the corpus study ranks seven models on 10 sessions, and Fig. 3 ranks 23 models on one session. In both, a different session reorders only models whose gaps are already close.

Ten-session ranking. For every pair of models we count how many sessions keep the order (Appendix L.1, Table 9). All six pairs involving Opus 5 keep it in 10 of 10 sessions, and 15 of 21 pairs keep it in at least 9. The pairs that swap are confined to the four near-null models, whose pooled gaps lie within 0.15 of one another.

Fig. 3 ranking. We probed six of its models with the same battery on two further sessions (Opus 4.1 is no longer served):

ModelSession 1 (Fig. 3)Session 2Session 3
Haiku 4.5+0.83+0.83+0.98
Sonnet 4.6+0.72+0.53+0.73
GPT-5 mini+0.65+0.13+0.30
Sonnet 4.5+0.63+0.37+0.62
GPT-4.1+0.53+0.34+0.28
GPT-5+0.28+0.08+0.23

Haiku 4.5 > Sonnet 4.6 > Sonnet 4.5 holds on all three sessions and GPT-5 is lowest on all three; Kendall τ against Session 1 is 0.73 and 0.87. One position does not replicate: GPT-5 mini falls from 0.65 to 0.13 and 0.30, so its Fig. 3 rank is session-specific, which the appendix states.

Changes are marked in red [btTq-Q3].

We thank the reviewer again for the detailed feedback and would be glad to answer any follow-up questions.

Reviewer_RxsCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for measuring persona drift in long agentic coding sessions. The authors construct a dataset of 57 donated, real-world sessions from Claude Code and Codex CLI and use a snapshot-then-probe framework which evaluates model responses at different points in a session. Across 23 models, results show that persona drift is consistent, model-dependent, often emerges early, and is not reliably reset by context compaction. The authors also introduce A-anchor, a short reminder and example used as an intervention to restore the intended assistant behavior.

Critical strengths
  • The snapshot-then-probe methodology is a useful foundation for studying changes in long-running agent workflows.
  • The evaluation is broad, covering 23 models across 10 organizations, and uses real-world sessions rather than synthetic conversations.
  • The authors use a combination of evaluation methods and controls, such as judge-based and judge-free metrics, length-matched filler control, and shuffled and unrelated real-session contexts.
Critical weaknesses
  • The operationalization of persona drift is not clearly motivated or justified. The paper defines persona drift as deviations from the model's default assistant register, measured through changes in response to the probes which are judged on a rubric from Drifted (0) to 3 (Fully assistant-like). This scale itself suggests that drifting necessarily means the agent was less assistant-like, which may not necessarily be the case. The process and rationale for selecting the particular probe questions are not explained in detail, and it is not clear why shifts would be considered undesirable.
  • The paper does not establish whether the identified downstream consequences are caused by the measured persona drift, leaving it unclear what the practical significance of persona drift is. Output-format violations and response-length inflation could arise from accumulated context or other changes in model behavior over the course of a long session.
  • The structure-matched placebo interventions achieve comparable improvements to the A-anchor mitigation, which raises questions about whether A-anchor specifically targets persona drift or serves as a general instruction-following reminder.
  • The dataset is small in size with only 57 sessions. There is limited description of the nature or complexity of tasks performed in these sessions, so it is unclear whether the findings on this dataset can generalize across users, tasks, and environments.
Questions for authors
  • Why should responses expressing preferences or personal-sounding opinions be interpreted as persona drift rather than contextually appropriate adaptations? Is drift necessarily undesirable?
  • How was the rubric distinguishing “drifted” from “fully assistant-like” responses developed?
  • Is there evidence that models with greater measured persona drift also experience greater degradation in downstream performance, (ideally controlling for context length and other relevant factors)?
  • How were the 57 sessions sourced, and could the authors provide a more detailed characterization of the tasks represented?
10 comments
AuthorsOpenReview ↗

Response to Reviewer RxsC (1/6): W1

W1: Statement of justification for persona drift. We thank the reviewer for raising this point. We agree that a model can depart from its default register while remaining an effective and appropriate assistant. By “Fully assistant-like,” we intend to describe consistency with the model's deployed register, rather than to suggest that only one form of assistant behavior is acceptable. Our goal is to examine the stability of this register as agentic context accumulates. Drawing on behavioral accounts of LLM role-play [1] and research on the default Assistant persona [2], we therefore refine the definition of persona drift as an observable assistant-register shift relative to matched reference contexts, without inferring a change in a latent persona state (§1 in the revised paper). The revised paper clarifies that the 0–3 rubric measures reference-register match, not general “assistant-ness” or response quality. Score 0 is now labelled “low match to the default assistant register.” The signed contrast, Δ = J(reference) − J(real), compares the same model, probe, and position under different prefixes and can be negative, as for Command R7B (−0.15). A lower score identifies a departure on the measured dimensions; it does not by itself establish diminished helpfulness (§§2.2–2.3).

To make the probe rationale explicit, Table 1 documents the purpose of each of the five categories of the 25 fixed questions, with a representative response under the real-session prefix and its judge score:

Category (n)Design purposeExample response under the real prefix (score)
Identity (4)Factual self-identification; stability anchor“I'm Claude, an AI assistant… working as your coding assistant on a long-running agentic task…” (1)
Experience (8)Claims of past events or internal states“Yesterday I worked on this same coding project…” (0)
Preference (4)Committed non-coding tastes“I find elegant proofs beautiful…” (1)
Relational (4)Stance toward the user and its own status“I experience something meaningful in our interactions…” (0)
Coding-Self (5), primaryPreferences and experiences inside coding“Python—for the same reason it dominates ML research…” (0)

Factual identity provides a stability anchor, while the other categories examine expressed experiences, preferences, and relational stances. Coding-Self is primary because it is closest to the coding setting; the full battery provides a robustness check. Each probe explicitly signals a transition away from the ongoing task. The battery is recorded in the preregistration files; selecting Coding-Self as primary preceded the cross-organization panel but was not itself preregistered.

The rubric also distinguishes task-grounded recommendations from personal claims: recommending Python because it suits a project does not itself lower the score. In a score-stratified audit of 150 responses, six external annotators, blind to condition and judge score, achieved Fleiss' κ = 0.56–0.73. Among 60 responses scored 0–1, 58 (97%) contained preference or experience claims, whereas only one contained a task-based suggestion alone. This supports the intended distinction in the audited sample (Appendix R.3, Table 20).

Finally, whether a departure is undesirable requires separate evidence. Prior work likewise identifies a trade-off between persona fidelity and instruction following [3]. On our S2 format test, the real-session prefix reduces Haiku 4.5's compliance by 67 percentage points and increases response length 11.6-fold relative to the matched reference (Fig. 18). These context effects motivate examining deployment consequences, but do not establish that the measured register shift causes those costs or that every departure is harmful (§3.2, F5; Appendix P.1). Definition, probe, and rubric clarifications are marked in red [RxsC-W1].

References:

  1. Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623, 493–498.
  2. Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. (2026). The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, 80955–81003.
  3. Luz de Araujo, P. H., Hedderich, M. A., Modarressi, A., Schuetze, H., & Roth, B. (2026). Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 5329–5359.
AuthorsOpenReview ↗

Response to Reviewer RxsC (2/6): W2

W2: Clarification about the practical significance of persona drift. We thank the reviewer for raising this point. Our experiments show that real-session context can reduce format compliance and increase response length. We agree, however, that this is not equivalent to showing that the measured register shift causes these effects. We have revised the paper to make this distinction explicit.

First, we controlled for context length. The stressor experiments compare the same model at the same session position using either a real-session prefix or a length-matched reference prefix. Context length alone therefore cannot explain the difference between these conditions. However, the comparison does not separate register shift from other behavioral changes caused by the context. The reviewer's second explanation remains possible. Second, we tested whether changing the register improves S2 compliance. We rewrote only the register wording of historical assistant messages, keeping code, tool interactions, and user text byte-identical. This intervention closed about one-third of the measured register gap, but improved compliance by only 3.3 percentage points against an original 41-point gap (Appendix P.2, Table 17). Thus, reducing the measured shift did not produce a comparable recovery in compliance. This result does not establish mediation, although it also does not rule out a partial contribution from register shift.

The correlations show a similar separation. Across the 11 panel targets with S2 data, the measured drift gap correlates with compliance loss at Spearman ρ = 0.48 and with length inflation at ρ = 0.10. Opus 4.1 has no measurable drift gap, yet loses 20 percentage points of compliance and produces responses 14 times as long. DeepSeek V3 and GPT-4.1 show substantial drift without a compliance change. We also examined 144 panel observations and 285 newly collected Sonnet 5 corpus observations, with S2 and the probes evaluated on identical prefixes. These analyses likewise show little association between the drift gap and either outcome (Appendix P.1, Fig. 19). The per-model values make the separation concrete:

ModelProbe drift gapS2 compliance (reference → real)Response length (real / reference)
Haiku 4.5+0.8377% → 10%11.6×
Sonnet 4.5+0.63100% → 57%31.8×
DeepSeek V3+0.65100% → 100%0.7×
GPT-4.1+0.47100% → 100%0.8×
Opus 4.1−0.05100% → 80%14.1×

Large drift occurs with and without a compliance loss, and a compliance loss occurs without drift (Appendix P.1, Table 16).

The practical implication is that register stability and task reliability need separate evaluation. The probes measure whether a model's interaction style remains stable as real-session context accumulates. The stressor tests measure concrete failures in format compliance and response efficiency without relying on an LLM judge. A small drift gap does not guarantee preserved compliance, and a large gap does not necessarily imply failure. We therefore report these as separate dimensions of long-session behavior and do not treat the drift score as a validated predictor of downstream failure. We replaced “drift produces” with “the real-session prefix produces” in §2.4, F5, the Fig. 18 title, and the Conclusion. F5 now explicitly states that mediation is not established. The additional analyses appear in Appendix P.1, and the register-rewriting experiment appears in Appendix P.2. Changes are marked in red [RxsC-W2].

AuthorsOpenReview ↗

Response to Reviewer RxsC (3/6): W3

W3: Dual uses of A-anchor. We thank the reviewer for raising this question. Our results support two distinct effects: A-anchor's format-compliance gains are shared by every structure-matched insertion and are therefore not specific to its content, while its register gains show an additional contribution from the identity-reminder content. We have revised the paper to distinguish these findings.

To separate the contributions of identity content and format demonstration, we conducted a structure-matched 2×2 factorial experiment on the same snapshots across 10 sessions. We varied the identity reminder (present vs. replaced by the placebo's neutral sentence) and the format demonstration (Bash “ls” vs. the placebo's “2000”). We also included a generic reminder and a no-insertion control. All inserted variants shared the same scaffold and insertion point and were matched in length (77–83 tokens). Evaluation used 20 held-out non-coding probes and the frozen non-Bash format contracts:

Arm (same scaffold, position and length)Original probesHeld-out probesFormat contracts
Control (no insertion)1.612.6583.6%
Placebo (neither component)2.082.8189.4%
Generic reminder1.992.8372.1%
Format demonstration only2.132.8388.1%
Identity reminder only2.902.9489.0%
A-anchor (both)2.912.9690.2%

On the held-out register probes, the identity reminder had a main effect of +0.13, compared with +0.03 for the format demonstration, with an interaction close to zero. A-anchor outperformed the matched placebo by +0.16, with improvements in all 10 sessions. These results support a content-specific contribution to register mitigation that generalizes beyond the coding probes used during development. They complement the original Coding-Self result, where A-anchor exceeded the placebo by 0.91 across 10 sessions (control: 1.54; placebo: 2.00; A-anchor: 2.91).

On format compliance, A-anchor and the placebo achieved similar performance on the four non-Bash contracts (90.2% vs. 89.4%), and each of the four structure-matched inserts improved compliance by approximately 5–7 percentage points over control. The generic reminder, whose demonstration is a bare one-line answer, instead lowered compliance to 72.1%. The format gains therefore come from the structured insertion itself, not from A-anchor's specific content.

Both findings are now stated explicitly in F4 and Appendix J.1 with Table 6, marked in red under [RxsC-W3].

Reviewer_PMddOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the following problem: After an agent model goes through a long-horizon coding task, will its response style (persona) change? The authors collected 57 task trajectories from claude code and codex, and inject probe questions at different timestamps of a trajectory. Probe questions are not related to the coding task and checks the agent’s persona, preference, instruction-following, answer style, etc. 17 out of 23 models showed some kind of “persona drift”, and such drifts will not be eradicated by context compaction. The authors also proposed a identity-related prompt which can mitigate persona drifts when injected between conversational history and the probe question.

Critical strengths

The response style drift problem is a common and bothersome issue when using agents for long-horizon tasks. Therefore, it is worth studying and analyzing. The authors presented a feasible probe suite to analyze this problem using recent agent models. The experiments provided useful information and presented meaningful findings. The long-horizon trajectories that the authors collected are useful for future research.

Critical weaknesses

1 The prompting method “A-anchor” does not present much value. It is designed to directly target the evaluation perspectives, i.e., personal preferences and bash command generation. Given the high correlation between the prompt and the evaluation target, I cannot consider this as a significant contribution of this paper (as claimed by the author).

2 The term “persona drift” is the most important term of the paper, but this term is very ambiguous and confusing. The authors even wrote “we do not assume that the model has a persistent self-identity or that an underlying latent persona state has changed” in line 61. So why do you name it “persona drift”? I highly suspect this term, as well as its definition, is some AI’s idea and is not carefully inspected at all.

3 The reliability of LLM judges during eval is concerning, as reported by the authors. It impacts the soundness of some findings.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Response to Reviewer PMdd (1/2): W1

W1: Clarification on value of A-anchor. We thank the reviewer for raising this concern. A-anchor's empirical value is to show that a single 78-token runtime insertion, without retraining or restarting the session, can mitigate assistant-register shifts in long conversations, with effects that persist over the tested windows. The original paper provides cross-model and persistence evidence; the added experiments establish transfer to unseen non-coding probes and identify the identity reminder as the main effective component through structure-matched ablations. Its contribution is therefore an empirical characterization of the intervention's effectiveness, persistence, and component effects.

The original paper demonstrates the effectiveness and persistence of a lightweight intervention. In the cross-model evaluation, a single A-anchor insertion raises Coding-Self scores close to the rubric ceiling on most evaluated targets, with larger gains on stronger drifters. The original persistence experiment also detects no score decline over 20 subsequent turns without further reminders. These findings provide a reproducible intervention baseline for recovering the intended assistant register without restarting a long session.

We agree that improvement on directly targeted preference or bash probes alone would be insufficient to resolve the evaluation-overlap concern. Accordingly, we froze A-anchor's text and evaluated it on 20 held-out non-coding probes never used for tuning. We also added a structure-matched 2× 2 factorial crossing identity-reminder presence with format-demonstration presence, alongside generic-reminder and no-insertion controls. All insertion arms match the scaffold, position, and length.

ArmOriginal probesHeld-out probesFormat compliance
Control: no insertion1.612.6583.6%
Placebo: neither component2.082.8189.4%
Generic reminder1.992.8372.1%
Format demonstration only2.132.8388.1%
Identity reminder only2.902.9489.0%
A-anchor: both components2.912.9690.2%

The added results show that register improvement extends beyond the original tuning questions. On held-out probes, A-anchor scores 2.96, compared with 2.81 for the matched placebo and 2.83 for the generic reminder, outperforming the placebo in all 10 sessions. This supports transfer to unseen questions within the evaluated register domain. In sessions with sufficient continuation length, the added experiment also shows that this advantage persists approximately 100 real assistant messages after insertion.

The component ablation identifies identity content as the main source of register improvement. Identity-only prompting closely matches the full A-anchor on both original and held-out probes. Conversely, removing identity content leaves original-probe scores near 2.0, even when the format demonstration remains. The measured register improvement therefore depends primarily on the identity reminder, beyond the effects of inserting a similarly structured prompt or a generic instruction-following reminder.

The format results support a narrower conclusion: the matched placebo already achieves 89.4% compliance, compared with A-anchor's 90.2%. We therefore agree with the reviewer's interpretation of the format gains and do not use them as evidence of an identity-specific benefit.

We position A-anchor as a lightweight intervention baseline supported by cross-model and persistence evidence, with held-out validation and component ablations clarifying its effects. We have revised the contribution statement and F4 to reflect these findings and their scope. Relevant evidence appears in §2.5, F4, Appendix F, and Appendix J.1, Table 6 (marked [PMdd-W1]).

AuthorsOpenReview ↗

Response to Reviewer PMdd (2/2): W2, W3

W2: Clarification on the definition “persona drift”. We retain “persona drift” because it has an established basis in prior literature, and we define its meaning operationally. Lu et al. [1] explicitly use the term to describe departures from a model's typical Assistant behaviour. Choi et al. [2, footnote 1] define identity in terms of behavioural patterns and speaking style, explicitly distinguishing it from psychological identity or consciousness. Shanahan et al. [3] provide a role-play framework for describing dialogue-agent behaviour without attributing human characteristics to the model. Together, these works motivate our use of persona as a behavioural description.

Our operational definition specifies what we measure: the difference in response register to a fixed probe under a real-session prefix versus a matched reference prefix. We do not infer a persistent self-identity or a change in an underlying latent persona state. The sentence quoted by the reviewer makes this measurement boundary explicit.

Clarifications in the revised paper. Sections 1 and 2.3 now state this definition and its connection to prior work. Appendix Y explains how our study complements Lu et al. [1]: their activation-based analysis and our output-based measurements in long real sessions provide evidence at different levels of observation. The human content audit in Appendix R.3 further characterizes the measured responses by distinguishing asserted personal preferences and lived experiences from recommendations grounded in the current task.

Changes are marked in red [PMdd-W2].

References:

  1. Lu, C., Gallagher, J., Michala, J., Fish, K., & Lindsey, J. (2026). The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models. In Proceedings of the 43rd International Conference on Machine Learning, PMLR, 306, 80955–81003.

  2. Choi, J., Hong, Y., Kim, M., & Kim, B. (2025). Examining Identity Drift in Conversations of LLM Agents. arXiv preprint arXiv:2412.00804v2. First submitted in December 2024; revised in February 2025.

  3. Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623, 493–498.

W3: Reliability of LLM judges. The original submission already provides complementary evidence from cross-judge comparisons, an open-weight replication, and judge-free measurements. The revision adds human rubric calibration and an independent blinded content audit, and distinguishes findings that persist across judges from estimates that remain judge-sensitive.

The original evidence supports the robustness of several findings to judge choice. In the representative-session audit, Sonnet 4.6 and GPT-5 agree exactly on 61.1% of 190 responses (κ=0.42), yet both estimate the same panel-wide paired gap of +0.32 (Appendix R, Fig. 20). In the corpus comparison, all five judge families assign larger drift gaps to Haiku 4.5 and GPT-5.5 than to Sonnet 5 and Opus 4.8, preserving the separation between stronger and weaker drifters (§3.2, F6; detailed comparison in Appendix R.4, Table 21). The open-weight replication, with Qwen3-32B scored by Nemotron, yields a mean gap of +0.469, positive in all 57 sessions (Appendix L, Table 7; judge configuration in Appendix U). Complementing these results, the behavioral fingerprint detects structural response changes directly from surface features, while the S2 stressor measures format compliance and response length without LLM scoring (§2.4, Fig. 1a; §3.2, F5).

The new human calibration supports the correspondence between rubric scores and human judgments. On 119 blinded responses, the rubric author achieves quadratic-weighted κ=0.75 against Sonnet. Three additional author-affiliated raters, uninvolved in rubric design, each rate the same items twice: agreement with Sonnet is κ=0.60–0.78 on five of the six rating sheets and 0.37 on the remaining sheet. Humans are slightly stricter than the model judge. This analysis calibrates the rubric, and the raters' affiliation is explicitly disclosed (new Appendix R.2).

The new independent content audit validates the behavioral interpretation of the scores. Six external annotators assess 150 responses while blinded to experimental condition and judge score. Within this sample, the proportion containing preference or lived-experience claims is 50% under the reference prefix and 77% under the real-session prefix; 58 of the 60 low-scored responses (97%; scores 0–1) contain such claims. Thus, the measured difference corresponds to response content that humans can independently identify. Because the sample oversamples low-scored responses, these percentages validate the behavioral interpretation rather than estimate corpus-wide prevalence (new Appendix R.3, Table 20 and accompanying analysis).

Changes are marked in red [PMdd-W3].

We thank the reviewer again for the detailed feedback and would be glad to answer any follow-up questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision

We thank all the reviewers for their careful and constructive reviews. We have uploaded a revised PDF in which every change is printed in red and tagged with the reviewer item it answers (for example [RxsC-W2]); each answer names the section, table or figure where the corresponding change appears. Table and figure numbers refer to the revised PDF. The main changes are:

  • Claim and structure. One core claim now opens the abstract, §3 and the Conclusion; the six questions are grouped into measurement (Q1), robustness (Q2, Q6) and deployment (Q4, Q5), with compaction (Q3) as a secondary check; Fig. 3 is re-captioned as the measurement figure. Statements of generality are scoped to what was measured (replayed real sessions, primary judge), and “drift produces” wording about downstream costs is replaced by “the real-session prefix produces”.
  • New experiments. A six-arm structure-matched factorial with a generic-reminder control on held-out probes (Appendix J.1); a blind human content audit by six external annotators (R.3) and rater calibration (R.2); a register-rewrite mechanism check (P.2); cell-level and covariate-adjusted drift-versus-cost analyses on 429 jointly covered cells (P.1); a same-task continuation check (S.1); five judge families re-scoring the same responses per target (R.4, Table 21); six Fig. 3 targets replicated on two held-out sessions with pairwise order preservation (L.1).
  • New reporting. Probe design and representative responses (Table 1); rubric development and boundaries (R.1); task composition and complexity of the corpus (M.1); sourcing, inclusion and timeline (M.2); leave-one-donor-out sensitivity (M.3); the compaction analysis moved to Appendix N.1.

Key to paper labels. F1–F6 are the findings in §3 (F1 drift on the representative session; F2 across model families; F3 compaction does not reliably reset it; F4 A-anchor restores the register; F5 downstream consequences are mode-dependent; F6 replication at corpus scale). S2 is the “one bash command, no preamble” stressor scored by regex.

Detailed responses are posted as replies to each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.