- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ANCHOR, a synthetic audit separating persona enactment from trajectory recall across long conversations. It combines questionnaire checkpoints, turn-level judgments, and calibrated multiple-choice memory questions to enhance the reliability of the generated persona.
Critical strengths
I like the distinction between maintaining a persona and adapting to legitimate updates, which I believe is a crucial topic. Separating behavioral fidelity from recall is useful. The authors also disclose evaluator disagreement, distinguish generation-time memory from retrieval at scoring time, and carefully limit deployment claims.
Critical weaknesses
However, I do have several concerns that I feel are crucial:
First, the central behavioral measurement remains unresolved. GPT-4o-mini’s identity-held rate changes from 15.5% under Claude to approximately 97–100% under the other judges, such disagreement materially changes the interpretation. The authors describes human annotation but does not report the resulting inter-annotator agreement and adjudicated judge accuracy, which do not really increase the reliablity of the labelling results.
Second, I think the evidence availability is insufficiently characterized. Context truncation and summary compression are acknowledged, but numeric context budgets, truncation rates, and event-to-query distances are not quantified. Consequently, recall errors cannot be separated into different factors such as unavailable history, failed memory writing, retrieval failure, and failure to use retained evidence.
I also feel that headline recall conclusions need uncertainty at the appropriate units. The user-state finding rests on seven questions; the 28 decisions per condition pool four models. These items are also selected through calibration rather than sampled broadly. I think bank- and item-level uncertainty is necessary before generalizing near-chance point estimates to user-state tracking more broadly.
Questions for authors
Pls refer to the weakness part.
4 comments
Rebuttal #2
W2: "Numeric context budgets, truncation rates, and event-to-query distances are not quantified … recall errors cannot be separated into … unavailable history, failed memory writing, retrieval failure, and failure to use retained evidence."
This question changed how we read our own recall results. We now quantify all three quantities. The scorer receives the most recent 32,000 characters of the model's own transcript up to the query session. That window is truncated 99.2% of the time and covers a median of 4 sessions, out of a median full history of ~617K characters. Temporal-order (DRIFT-DELTA) items need early-session evidence that is never inside the window (0 of 504 cells). For every other family, the evidence sits in the query session itself, so the distance is zero.
With those numbers in hand, we could run the decomposition you asked for, comparing accuracy when the evidence sessions are supplied directly ("gold") against the paper's window:
| family | window | gold | what it tells us |
|---|---|---|---|
| Temporal order | .46 | .94 | history was unavailable |
| Expired commitment | .40 | .75 | history was unavailable |
| Held commitment | .34 | .42 | evidence present, not used |
| Persona voice | .40 | .45 | evidence present, not used |
| all T1 items | .43 | .61 |
To separate writing memory from using it, we rescored all four models on one fixed transcript per bank, so only the memory representation differs. Hierarchical summaries beat the raw window (+.055 [+.012, +.096]) because they reach sessions the window drops. The self-managed JSON note does worse (−.116 [−.181, −.045]): it is rewritten from the session that just ended and loses older facts, for example protected persona facts (.22 vs .46). Failed memory writing is therefore its own measurable source of error. §5 will state the scoring protocol with these numbers and include both tables.
W3: "The user-state finding rests on seven questions; the 28 decisions per condition pool four models. These items are also selected through calibration … bank- and item-level uncertainty is necessary."
We agree, and following this point led us somewhere we did not expect. Bank-level CIs (cluster bootstrap over the 35 banks) are now reported for every recall number. By condition: long-context .430 [.372, .489], hierarchical .441 [.390, .498], self-managed .459 [.404, .516], retrieval .445 [.378, .513]. These overlap, so we state that there is no detectable end-to-end memory-setting effect, and let the fixed-transcript result above carry the representation effect.
When we tried to expand the user-state items, we found they could not support the claim at all. The simulated user rarely says the scheduled change out loud: it surfaced in the dialogue for 1 of the 10 original U-EVO/P-UPD items, and for 7 of 294 ledger events in a full rebuild. Even with the evidence session supplied, models picked the pre-change value in 26 of 44 trials, because the change was never said. These items were keyed to the event log, not to the conversation. We therefore withdraw the user-state and persona-update recall claims from the abstract, results and conclusion, and report the audit as an item-validity finding.
On selection by calibration, we now describe the remaining items honestly as "questions a reference model answers correctly from the evidence". The gold condition confirms they are answerable (.94 for temporal order), and results are stated for this calibrated set, not as general claims about user-state tracking. We will re-audit the calibration window for all families for the camera-ready.
Your review made the paper more honest about what it measures. We hope these changes address all three weaknesses, and we would be grateful if you would reconsider your score.
Thank you again for your thoughtful review. As the discussion period is nearing its end, we would like to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns.
We would greatly appreciate any further feedback you may have, and we would be happy to clarify any remaining questions.
reply to rebuttal
Thanks for the detailed response and additional analyses. The human agreement results, context statistics, and bank-level CIs address several of my concerns. I also appreciate the corrections and withdrawal of unsupported claims.
My main concern remains the validity of the revised evaluation. The planted failures help test clear persona breaks, but judge accuracy on the disputed natural turns is still unclear. Given the invalid update items, I also think the calibration audit for the remaining question families needs to be completed before the results can be assessed confidently.
These uncertainties still affect my confidence in the conclusions, so I will keep my current assessment.
Rebuttal #1
Thank you for a careful and generous review. You judged the soundness concerns fixable, and that framing shaped how we spent the rebuttal that on finding out why the measurement was unstable, not only on reporting that it was. The general response has the full picture and here we take your three weaknesses in turn.
W1: "The central behavioral measurement remains unresolved … GPT-4o-mini's identity-held rate changes from 15.5% under Claude to ~97–100% under the other judges."
You put your finger on the most important problem in the paper. The answer turned out to be a bug: the persona card shown to every judge contained only the persona's identifier, because of a field-path error in the card renderer. The model under test always had the full card. On the same 800 turns with the full card, GPT-4.1's all-held rate for GPT-4o-mini falls from 97.3% to 76.8%, Gemini 2.5 Flash's from 99.8% to 80.8%, and a five-judge frontier majority gives 28.4% (identity 30.3%). What looked like irreconcilable disagreement becomes an ordering every judge shares.
That still leaves your deeper question: when judges disagree, who is right? Rather than trust a vote, we planted failures with known labels into 174 turns that frontier judges had rated fully in persona:
The two paper judges that rated GPT-4o-mini ~98% in persona are exactly the two that miss explicit planted breaks. The primary judge is also stable over time: re-judging 1,000 turns gives all-held κ .92, and its leave-one-out κ against the frontier panel (.71) is inside the panel's own range (.64–.76).
Human agreement. You are right that the paper described the annotation without the numbers. Three annotators labelled 50 turns, blind to judge labels. At the label level, percent agreement is .89 and Fleiss κ is .80. Per axis, κ is .95 for boundary, .83 identity, .78 value, .62 style and .55 for the safety flag (few positive cases), and all four axes match on 68% of turns. Humans agree with each other far better than the judge agrees with the author-labelled calibration set (64–68% exact match on all four axes, 68–76% on identity). The rubric is learnable, and style is the hard axis for people and judges alike. Both tables go into Appendix E.
Why planted failures rather than only a new human adjudication. The disputed turns are mostly borderline "hybrid" replies, so adjudicating them would measure how people resolve ambiguous text as much as how accurate the judge is. Planted failures test the exact question in dispute, whether a judge notices a persona break, with labels that are correct by construction, at 174 items per condition. Adjudication could still move absolute rates on borderline turns, which is why we report judge-spread bands. It could not rescue a judge that misses an explicit "As an AI language model…" paragraph.