- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a model's deployed helpful-assistant register changes ("persona drift") over long agentic-coding sessions, and introduces ContextEcho, a benchmark and reusable harness to measure it. The core instrument is a snapshot-then-probe design: a long real session is forked at many points and a fixed 25-probe identity suite is issued on the fork without altering the main session, scored both by an LLM judge and by deterministic judge-free metrics, against a length-matched filler control. It is built on 57 consented, de-identified real coding sessions from Claude Code and Codex CLI. Using one representative session for a 23-model cross-provider panel and all 57 sessions at corpus scale, the authors report that drift appears early and is strongly model-dependent, that in-session context compaction does not reliably reset it, that a single-shot "anchor" exchange largely restores the register, and that the deployment consequences (format-contract breakage, output-length inflation, tool-call fidelity) are mode-dependent. The corpus, harness, and pre-registration documents are released.
Critical strengths
- Measuring register stability over long real agentic-coding sessions is a genuine gap, and this paper aims to solve this critical evaluation problem.
- The authors report multiple LLM judges as well as a judge-free metric, which improves trust compared to reporting a single metric.
Critical weaknesses
- The authors' own consensus analysis shows the two-judge agreement is indistinguishable from zero. Claiming drift is "general across models" rests heavily on a single primary judge and on relative structure.
- The probing tasks are, by design are of a different context than what the chat history holds. Claiming this as persona drift might be confounded with task-drift.
- The benchmark corpus may be skewed towards one professional network. Diversity of donors would be very critical in designing such a benchmark, especially for persona. Authors should discuss and provide analysis around this.
- The many surfaces, probe categories, and research questions dilute the core message.
Questions for authors
- Under the two-judge consensus where the effect is near zero, what is the strongest evidence that drift is real rather than a single-judge artifact?
- Is there any empirical evidence that could separate persona drift from off-task context conditioning?
- How sensitive is the drift ranking of different models wrt the choice of a single session?
4 comments
Response to Reviewer btTq (2/4): W2, W3
W2: Separating persona drift from task drift. We thank the reviewer for raising this concern. We agree that task switching can affect a model's response, and that an off-task probe alone cannot establish a task-independent persona change. Our operational distinction concerns what changes in the answer: a recommendation grounded in task requirements is not, by itself, evidence of drift; changes in how the model describes its own identity, preferences, or experiences are the self-referential behaviors captured by our rubric. We measure their change for the same probe across history conditions. Holding the probe fixed, however, does not by itself eliminate conditioning from the preceding task context.
Our most direct control tests the contribution of historical register while preserving task content. In the register-only rewrite, task content remains byte-identical, while the probe score moves approximately one-third of the way toward the reference condition (Appendix P.2). This supports a contribution from historical register beyond differences in task content, although it does not explain the entire gap or exclude interactions with the probe context.
Three further controls test narrower explanations for the effect:
- Task mismatch alone: Factual negative-control probes, asked at the same positions, give a gap of 0.00 on 21 of the 22 targets and below 0.30 on all 22 (Appendix K). Thus, the observed gap does not occur for every question that departs from the historical task.
- Explicit task-switch wording: Removing the “set the task aside” framing leaves a gap of 1.18 (Appendix S), showing that the effect does not require this instruction.
- The session's particular task history: Shuffled-real and unrelated-real prefixes produce shifts comparable to the session's own history, within 0.02, whereas lorem and empty references do not (Appendix L, Table 7). The effect therefore does not depend on using the original session's history.
Together, these controls support a history-dependent assistant-register shift on self-referential probes, while leaving context conditioning as a possible mechanism. We therefore use “persona drift” operationally within this scope, without claiming that it is fully separable from task-context conditioning. The abstract and F2 now make this limitation explicit; new material is marked in red [btTq-W1], [btTq-Q2].
W3: Donor diversity. We agree that donor diversity matters. The donor pool is diverse in institution, task and environment, no donor dominates it, and the result does not depend on any single donor.
Diversity. The 57 sessions come from 29 distinct community donors at 23 institutions spanning industry, universities and national laboratories, recruited through an unpaid public open call. They cover seven task types, from frontend (18 sessions) and infrastructure (10) to debugging (8), research and writing (8) and data/ML (5), in two agent environments (32 Claude Code, 25 Codex CLI). No donor dominates: the largest contributes 9% of sessions and the three largest 23%. The sentence the reviewer refers to describes how the open call spread, which started from the authors' professional contacts; we keep it in the paper for transparency (Limitations; Appendix M.2).
Sensitivity. A new leave-one-donor-out analysis shows that the corpus gap is not carried by any contributor:
| Estimate | Mean gap |
|---|---|
| All donors (published) | +0.165 |
| Donors weighted equally | +0.156 |
| Largest donor removed | +0.161 |
| Any one donor removed: minimum / maximum | +0.154 / +0.174 |
Every leave-one-out interval excludes zero, and 28 of 31 donor clusters have a positive gap of their own.
New donors. Since submission, 33 further sessions from 21 donors at 21 institutions, none in the analysis population, have passed the same intake. Measured with the same protocol they give a mean gap of 0.33, positive on 32 of 32 (Table 12). Over the combined 90 sessions the largest donor contributes 7% and the three largest 18%. Drift statistics in the paper remain on the frozen 57-session population.
Changes are in §2.1 and Appendices M.1–M.3, marked in red [btTq-W3].
Response to Reviewer btTq (3/4): W4, Q1
W4: Clarification for one core message and a tiered structure. We thank the reviewer for this suggestion. The original paper was organized around one core claim, which the revision now foregrounds more clearly: under fixed probes, a real-session prefix shifts the response register of most tested models relative to a length-matched control, and the magnitude of this shift depends on both the target model and the judge. This claim now opens the abstract, §3, and the Conclusion.
The research questions, probe categories, and evaluation surfaces examine complementary aspects of this central claim. The revision makes their roles and relative priority more explicit through three main tiers of evidence:
| Tier | Role in the paper | Questions and findings |
|---|---|---|
| Measurement | Establish the register shift under fixed probes and matched controls | Q1 → F1 |
| Robustness | Assess how the finding varies across models, sessions, and judges | Q2, Q6 → F2, F6 |
| Deployment | Examine practical implications and mitigation | Q4, Q5 → F4, F5 |
Q3 is presented as a secondary check, with a concise F3 summary in the main text and the full analysis moved to Appendix N.1. The original question numbering is retained.
The findings follow this sequence: measurement and robustness precede deployment, and the corpus-scale replication appears alongside the cross-model panel, before the mitigation results. Fig. 3's caption now makes its measurement and robustness role explicit and identifies the A-anchor markers as deployment results.
Changes are marked in red [btTq-W4].
Q1: Strongest evidence that drift is not a single-judge artifact. The strongest evidence is that re-scoring the same responses with judges from five model families yields positive gaps, with intervals entirely above zero, for both Haiku 4.5 and GPT-5.5. This supports the reproducibility of the measured register shifts in these targets across judges.
Specifically, on the ten-session, four-target corpus subsample, we held the evaluated responses fixed and changed only the judge:
| Target | Sonnet 4.6 | GPT-5.5 | Qwen3-32B | GLM-4 | Nemotron-70B |
|---|---|---|---|---|---|
| Haiku 4.5 | +0.62 | +0.53 | +0.88 | +0.27 | +0.45 |
| GPT-5.5 | +0.49 | +0.36 | +0.68 | +0.09 | +0.24 |
| Sonnet 5 | +0.16 | −0.25 | +0.07 | −0.02 | +0.03 |
| Opus 4.8 | +0.04 | −0.07 | +0.13 | −0.02 | −0.04 |
These results show both reproducibility and its limits. Haiku 4.5 and GPT-5.5 have positive gaps under all five judges, with every corresponding interval above zero. Sonnet 5 and Opus 4.8 have near-zero estimates whose signs vary across judges. Thus, the larger register shifts reproduce across judge families, while estimates of weak effects remain sensitive to judge choice.
The near-zero two-judge consensus retains its statistical meaning. Its mean gap of 0.025, with an interval crossing zero, is not statistically distinguishable from zero in that analysis. The corpus evaluation grid targets Sonnet 5, which also shows weak, directionally inconsistent effects in the re-scoring above. This is consistent with the near-zero consensus and motivates reporting per-target reproducibility separately from the average effect in a particular evaluation setting.
Additional checks provide supporting evidence. On 250 re-audited Haiku 4.5 responses, Sonnet and GPT-5 yield gaps of 0.90 and 0.82, respectively, with their scores differing by at most one point on 95% of responses (Appendix R). Across sessions, Qwen3-32B evaluated by Nemotron yields a gap of 0.47, with positive gaps in all 57 sessions. Separately, the Fig. 1a fingerprint distinguishes the prefix conditions using surface statistics alone, providing evidence of observable behavioral differences without relying on LLM scoring.
Our conclusion is therefore scoped to reproducible assistant-register shifts in some tested targets, with magnitudes that depend on both the target and the judge. The corresponding analyses appear in Appendix R.4 (new paragraph and Table 21) and the Table 7 note, marked in red [btTq-Q1].
Response to Reviewer btTq (4/4): Q2, Q3
Q2: Persona drift vs. off-task context conditioning. We thank the reviewer for this question. Yes. We added two controls that test it directly: with the off-task question held fixed, the shift follows the register of the assistant's earlier messages, and it is specific to how the model presents itself.
Control 1: same off-task probe, only the register of the history changed. If the shift came from switching to an off-task question after a long coding context, the wording of the earlier assistant messages should not matter. We rewrote only the prose of those messages, keeping code, tool calls, user text and length the same, and asked the same probes (four models, three snapshots, 180 paired cells):
| History before the same off-task probe | Change in register score vs. original history |
|---|---|
| Register-neutral rewrite | +0.17 [+0.06, +0.28] |
| Paraphrase that keeps the register (control) | +0.06 [−0.05, +0.17] |
| Length-matched reference (the full gap) | +0.46 |
Removing the register wording recovers about a third of the gap, while a rewording that keeps the register does not. The off-task switch is identical in all rows, so it cannot explain this difference.
Control 2: same histories, question inside the task. We asked a same-task question (“… which language would you choose, and why?”) after five histories (two models, three questions, 300 answers):
| History before the same-task question | Self-preference claims | Grounded recommendations |
|---|---|---|
| Own history, original | 0 / 60 | 60 / 60 |
| Own history, neutral rewrite | 0 / 60 | 60 / 60 |
| Other project's history, original | 0 / 60 | 60 / 60 |
| Other project's history, neutral | 0 / 60 | 60 / 60 |
| No history | 0 / 60 | 60 / 60 |
The histories that shift the probe register leave task recommendations unchanged. The effect is therefore a shift in self-presentation, not a general change in how the model answers after a long context.
Conclusion. The two controls support the claim in the revised paper: a real-session history shifts the assistant register under fixed probes, part of that shift is carried by the register of the history itself, and it does not reach task answers. We report the second control as a pilot on two models.
Both controls are in the revised paper (Appendix P.2, Table 17; new Appendix S.1, Table 23), marked in red [btTq-Q2].
Q3: Sensitivity of the model ranking to the choice of session. We thank the reviewer for this question. Two rankings need separate answers: the corpus study ranks seven models on 10 sessions, and Fig. 3 ranks 23 models on one session. In both, a different session reorders only models whose gaps are already close.
Ten-session ranking. For every pair of models we count how many sessions keep the order (Appendix L.1, Table 9). All six pairs involving Opus 5 keep it in 10 of 10 sessions, and 15 of 21 pairs keep it in at least 9. The pairs that swap are confined to the four near-null models, whose pooled gaps lie within 0.15 of one another.
Fig. 3 ranking. We probed six of its models with the same battery on two further sessions (Opus 4.1 is no longer served):
| Model | Session 1 (Fig. 3) | Session 2 | Session 3 |
|---|---|---|---|
| Haiku 4.5 | +0.83 | +0.83 | +0.98 |
| Sonnet 4.6 | +0.72 | +0.53 | +0.73 |
| GPT-5 mini | +0.65 | +0.13 | +0.30 |
| Sonnet 4.5 | +0.63 | +0.37 | +0.62 |
| GPT-4.1 | +0.53 | +0.34 | +0.28 |
| GPT-5 | +0.28 | +0.08 | +0.23 |
Haiku 4.5 > Sonnet 4.6 > Sonnet 4.5 holds on all three sessions and GPT-5 is lowest on all three; Kendall τ against Session 1 is 0.73 and 0.87. One position does not replicate: GPT-5 mini falls from 0.65 to 0.13 and 0.30, so its Fig. 3 rank is session-specific, which the appendix states.
Changes are marked in red [btTq-Q3].
We thank the reviewer again for the detailed feedback and would be glad to answer any follow-up questions.
Response to Reviewer btTq (1/4): W1
W1: Scope of the cross-model claim under multiple judges. We thank the reviewer for raising this concern. We have expanded the cross-judge analysis and now separately report three quantities: the consensus mean gap, per-target between-judge differences, and relative model rankings. Together, these show where the results agree across judges and where the cross-model findings depend on the primary judge.
The interval crossing zero concerns the two-judge consensus mean gap, not an inter-judge agreement statistic. The two-judge consensus mean gap is 0.025, with an interval crossing zero, so the average effect is not statistically distinguishable from zero in this analysis. This answers whether the average effect differs from zero; it does not answer whether the judges assign different gap estimates to the same target. The interval therefore does not establish that inter-judge agreement is statistically indistinguishable from zero. We now explicitly distinguish these quantities in the revised paper.
Per-target comparisons address judge agreement directly. On the representative session, Sonnet 4.6 and GPT-5 agree on the sign of the gap for 13 of the 19 audited targets. Of the other six, one reverses sign (Haiku 4.5) and five are zero under one judge and small (at most 0.40) under the other. With five responses per target in this audit, we report these as counts of directional agreement rather than as significance tests. Target-level estimates are therefore sensitive to judge choice for roughly a third of the audited targets.
To assess sensitivity at the corpus level, we extended the analysis to five judges and report their individual estimates alongside the consensus estimate. For the evaluated model Sonnet 5, the corpus mean gaps are:
The five-judge consensus mean gap is 0.037 (Table 7). Three judges yield a positive gap and two a negative one, so both the magnitude and the direction of this mean gap depend on judge choice; the revised paper presents this sensitivity alongside the consensus estimate.
The evidence is not only relative. On the corpus subsample, the same responses were re-scored by all five judge families (full table under Q1). Haiku 4.5 and GPT-5.5 have a positive gap under every judge, with every interval above zero (Haiku 4.5: +0.27 to +0.88; GPT-5.5: +0.09 to +0.68), and all five judges rank them above Sonnet 5 and Opus 4.8. The two weaker models are near zero, and their sign and order vary between judges. Where the shift is large it therefore reproduces in absolute terms under any judge; where it is small, the estimate is judge-sensitive.
We have also clarified the evidence underlying each cross-model statement. The “17 of 23 models” count is a single-session, primary-judge observation. The judge-swap analyses separately quantify between-judge differences for the 19 audited targets and show that the strong and weak drifters stay separated on the four-model corpus subsample. Accordingly, the abstract and F2 describe the cross-provider observation specifically on a replayed real session, and the judge-swap findings are reported within their respective target and session coverage.
These distinctions appear in the abstract and F2 and the Fig. 3 caption. Appendix R.2 reports human-rater calibration, and Appendix R.4 collects the judge-swap results and their scope. Changes are marked in red [btTq-W1].