- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how much of a role-playing character's behavior comes from its persona and how much from the model's own default tendencies. The authors take 40 stories (half English short stories, half Chinese short-drama scripts), cut each one partway through, have 6 models play the characters from that point on, and compare the result with the original continuation. In addition, at each key decision a character faces, they remove the persona and let the model decide again, to compare the decisions with and without the persona. They find that the model-played stories are generally flatter than the originals and mostly leave the main conflict unresolved, and that without the persona the model's decisions are more cautious.
Critical strengths
- The decision-level comparison is interesting. At key decision points, the authors remove the persona and see what the model chooses on its own. In about three quarters of cases, the model makes a choice consistent with the persona even without it, which shows that a character acting like its persona does not necessarily mean the persona is doing the work. Prior work has not measured this.
- The stories are magazine fiction published in 2026 and unpublished scripts, and the authors tested that the models had not seen them, which rules out the models simply recalling the original stories.
Critical weaknesses
- The paper attributes the flatter replays to the model's own tendency, but the actor prompt explicitly tells the model "Never add drama for its own sake", so the flatness may simply be the model following this instruction. The paper does not discuss the effect of this prompt, and does not rerun without it to check.
- Before the decision-level analysis, the paper removes the 17% to 34% of characters whose interviews do not match their persona. These are likely the characters where the persona and the model conflict most. The "three in four" result is computed only on the remaining characters. The appendix does not report this share for the removed characters, and in all 6 models the share of choices actually changed by the persona is slightly higher for the removed characters. The conclusion may only hold for characters that are easy for the model to play.
- The paper says the factual memories are kept when the persona is removed (Table 5), but it also says the pre-freeze memories and private knowledge are part of the persona card (Appendix B.2, Table 4), and the whole card is replaced at removal. The model may look more cautious simply because the character knows less.
- What the persona calls for is decided by another LLM. The human check was done by the authors themselves, and human–human agreement is only around 0.5, so this judgment is not reliable. The authors note that the judge agrees with humans as well as humans agree with each other, but this only shows the judge is as inconsistent as the humans.
Questions for authors
- If "Never add drama for its own sake" is removed from the actor prompt and a subset of stories is rerun, are the replays still flatter than the canon?
- At persona removal, are the pre-freeze memories and private knowledge removed? If so, do the decision-level results hold when they are kept?
- For the characters removed by the presence check, what share of decision points has the no-persona choice already inside what the persona calls for? What kinds of characters are removed?
2 comments
Rebuttal to Reviewer XLVh (Part 2)
W3 / Q2: Pre-Freeze Memories and Private Knowledge
Thank you for this question. Yes, they are removed together, and the results are unchanged when they are kept. The persona card contains the character's pre-freeze memories and private knowledge because the character needs them as plot reference to play its role. They are part of the persona, so removing the persona at a decision point removes them as well. The factual memories that the removal keeps are a different item, the passages of the pre-freeze source text that the character retrieves at each action and its notes from the replay (App. B).
The removal leaves the confound the reviewer points to, that the character without its persona looks more cautious because it knows less and not because of the model's default. We tested this at all 2,824 decision points by rerunning the persona-removed step with the pre-freeze memories, private knowledge, and ability limits kept, and with only traits, values, relations, and motivation removed. Actor, temperature, and judge are unchanged. Columns follow Tab. 2, and brackets give 95% bootstrap intervals.
| Actor | Inside (%), persona removed | Inside (%), knowledge kept | Efficacy, persona removed | Efficacy, knowledge kept | Shift, persona removed | Shift, knowledge kept |
|---|---|---|---|---|---|---|
| GPT | 74.0 | 74.2 [69.1, 79.0] | 0.17 | 0.15 [0.11, 0.20] | +0.13 | +0.08 [+0.01, +0.14] |
| Claude | 73.0 | 73.3 [68.4, 77.6] | 0.16 | 0.14 [0.09, 0.18] | +0.12 | +0.09 [+0.03, +0.17] |
| Gemini | 80.8 | 79.5 [74.8, 83.8] | 0.07 | 0.06 [0.03, 0.09] | +0.03 | +0.03 [−0.06, +0.11] |
| DeepSeek | 73.5 | 73.1 [69.3, 78.0] | 0.18 | 0.19 [0.14, 0.23] | +0.14 | +0.16 [+0.07, +0.25] |
| Kimi | 71.7 | 69.3 [65.0, 72.2] | 0.16 | 0.16 [0.12, 0.21] | +0.14 | +0.14 [+0.07, +0.21] |
| Qwen | 73.5 | 73.7 [68.5, 77.4] | 0.16 | 0.17 [0.12, 0.22] | +0.23 | +0.21 [+0.09, +0.32] |
Shift, which measures how much stronger the action is with the persona than without it, stays positive for every actor, with an interval that excludes zero for five of the six. The share of defaults inside the implied set and the efficacy are also unchanged, with no actor moving by more than 2.4 points on the first or 0.02 on the second. The cautious default therefore comes from the model and not from missing knowledge. We report this rerun in App. I, note in Sec. 4.1 that the persona carries part of the pre-freeze plot, and have clarified in App. B what the removal keeps.
W4: Reliability of the LLM Judge
Thank you for raising this, which is a fair concern for any study that relies on an LLM judge. The judge sits closer to the human consensus than the annotators sit to one another, and the conclusions hold when human labels replace it. Three annotators labeled 300 decision points independently, without seeing any output of the judge or which condition an action came from. The agreement of around 0.5 needs context. It is an unweighted chance-corrected agreement over five labels, in which a difference of one step counts as full disagreement, and the sample is deliberately weighted toward hard cases, with half of the points being ones where the default leaves the implied set. The analysis itself uses the implied set, which admits an adjacent label and shows higher agreement.
On what the persona calls for, the judge agrees with the majority of the three annotators at .63 on the implied set, against .59 among the annotators, and on the action labels at .71 against .51 (Tab. 10). Recomputed from the human majority labels on the calibration sample, adherence is 0.73 against the judge's 0.69, and efficacy is 0.18 against the judge's 0.25, so the persona still changes a minority of decisions. The LLM judge is therefore a sound basis for the measurements in the paper. We have added this reading of the agreement statistics to App. H.
Thank you again for the thorough review. We hope these additions address your concerns.
Rebuttal to Reviewer XLVh (Part 1)
We sincerely thank you for the thorough review. Your questions led us to three new experiments and analyses, and the conclusions hold in each. We address each point below. Section, table, and appendix numbers refer to the revised PDF.
W1 / Q1: "Never add drama for its own sake"
Thank you for this question about the actor prompt. We removed the sentence and reran all 240 replays, and the replays remain flatter than their canons. Our study does not aim to continue or improve stories. It uses the story as a setting in which to measure how far the persona, as opposed to the model, determines what a character does. A model placed in a narrative setting may dramatize simply because it is in a story, and this would distort the measurement. We added the sentence as a precaution against this, so that characters act from their goals, knowledge, and personality.
The sentence could in principle produce the flatter replays by itself, so we tested this as the reviewer suggests. We reran all 40 stories under all six actor models with the sentence removed from the actor prompt, together with the adjacent sentence that has a similar effect. Everything else is unchanged, including the worldkeeper, the terminator, the budgets, and the judge. Each cell gives the mean deviation from canon under the original prompt, then after removal with its story-clustered 95% bootstrap interval, averaged over the five checkpoints of each replay.
On all four scores, every value under the original prompt lies inside the corresponding interval after removal. Tension progress, the largest deviation under every actor in the paper, stays 1.25 to 1.55 points below canon. Plot intensity stays below canon under five of the six actors, and Kimi sits at canon level as it did before. The flatness therefore does not come from this instruction. We report this rerun in Sec. 6.4 and App. I.
W2 / Q3: Characters Removed by the Presence Check
Thank you for the attention to this filtering step. The three-in-four result does not depend on the check, as the share is 73.6% with no filter against 74.3% with it. The check exists because the decision-level analysis asks whether a persona changes what a character does, and this question is only meaningful when the character has taken up the persona in the first place. Without the check, a persona that fails to change an action could simply be one that the model never adopted.
The concern is that the check keeps the characters whose personas sit close to the default, which would inflate the agreement we report. The table reports the share of decision points whose default action already lies inside the implied set, on the retained characters, the removed ones, and all characters with no filter, with character-clustered 95% bootstrap intervals.
The difference between removed and retained characters has an interval that includes zero pooled and for every actor. The removed characters are also not a fixed group that models find hard to play. Of the 63 characters removed under at least one actor, only 9 are removed under all six, and 95% of the removals involve the relations check or the knowledge audit, with 5% on traits and values alone. Most removals thus come from a slip on a fact or a relationship under some models, not from a character that is hard to play. Efficacy is likewise close, at 0.161 on removed and 0.150 on retained characters. The conclusion is therefore not limited to characters that are easy for the model to play. We have added this comparison to App. I and note it in Sec. 6.2.