acceptodds
Under review as a conference paper at ICLR 2027

ANIMASK: What the Model Contributes to Role Play in Simulated Story Worlds

Abstract

When a language model plays a character, the observed behavior reflects both the assigned persona and the default dispositions of the actor model itself. Existing evaluations test persona fidelity or model defaults in isolation, but neither says, at a specific choice with consequences, what the persona changed and what the model's default kept. We introduce ANIMASK, a simulation framework that freezes books and scripts into story worlds whose characters act on their own motivations and replays each story from its freeze point. We hold out the author's continuation as a human reference, verify through in-story interviews that each persona remains present, and at every decision point compare the character's action with what the model produces when the persona is removed. Across 40 stories, 6 actor models, and 3,846 decision points, the replays converge away from their canons in one shared direction, toward flatter, cooler stories that leave their tensions open. The personas stay present and obeyed throughout. On three choices in four the model's default already falls inside what the persona accepts, and where the two diverge the model is the cautious one, holding where the persona would press. The persona guarantees who the character is, and the model sets how far the character will go.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
role play, persona, LLM evaluation, LLM agents, social simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XLVhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much of a role-playing character's behavior comes from its persona and how much from the model's own default tendencies. The authors take 40 stories (half English short stories, half Chinese short-drama scripts), cut each one partway through, have 6 models play the characters from that point on, and compare the result with the original continuation. In addition, at each key decision a character faces, they remove the persona and let the model decide again, to compare the decisions with and without the persona. They find that the model-played stories are generally flatter than the originals and mostly leave the main conflict unresolved, and that without the persona the model's decisions are more cautious.

Critical strengths
  1. The decision-level comparison is interesting. At key decision points, the authors remove the persona and see what the model chooses on its own. In about three quarters of cases, the model makes a choice consistent with the persona even without it, which shows that a character acting like its persona does not necessarily mean the persona is doing the work. Prior work has not measured this.
  2. The stories are magazine fiction published in 2026 and unpublished scripts, and the authors tested that the models had not seen them, which rules out the models simply recalling the original stories.
Critical weaknesses
  1. The paper attributes the flatter replays to the model's own tendency, but the actor prompt explicitly tells the model "Never add drama for its own sake", so the flatness may simply be the model following this instruction. The paper does not discuss the effect of this prompt, and does not rerun without it to check.
  2. Before the decision-level analysis, the paper removes the 17% to 34% of characters whose interviews do not match their persona. These are likely the characters where the persona and the model conflict most. The "three in four" result is computed only on the remaining characters. The appendix does not report this share for the removed characters, and in all 6 models the share of choices actually changed by the persona is slightly higher for the removed characters. The conclusion may only hold for characters that are easy for the model to play.
  3. The paper says the factual memories are kept when the persona is removed (Table 5), but it also says the pre-freeze memories and private knowledge are part of the persona card (Appendix B.2, Table 4), and the whole card is replaced at removal. The model may look more cautious simply because the character knows less.
  4. What the persona calls for is decided by another LLM. The human check was done by the authors themselves, and human–human agreement is only around 0.5, so this judgment is not reliable. The authors note that the judge agrees with humans as well as humans agree with each other, but this only shows the judge is as inconsistent as the humans.
Questions for authors
  1. If "Never add drama for its own sake" is removed from the actor prompt and a subset of stories is rerun, are the replays still flatter than the canon?
  2. At persona removal, are the pre-freeze memories and private knowledge removed? If so, do the decision-level results hold when they are kept?
  3. For the characters removed by the presence check, what share of decision points has the no-persona choice already inside what the persona calls for? What kinds of characters are removed?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer XLVh (Part 1)

We sincerely thank you for the thorough review. Your questions led us to three new experiments and analyses, and the conclusions hold in each. We address each point below. Section, table, and appendix numbers refer to the revised PDF.


W1 / Q1: "Never add drama for its own sake"

Thank you for this question about the actor prompt. We removed the sentence and reran all 240 replays, and the replays remain flatter than their canons. Our study does not aim to continue or improve stories. It uses the story as a setting in which to measure how far the persona, as opposed to the model, determines what a character does. A model placed in a narrative setting may dramatize simply because it is in a story, and this would distort the measurement. We added the sentence as a precaution against this, so that characters act from their goals, knowledge, and personality.

The sentence could in principle produce the flatter replays by itself, so we tested this as the reviewer suggests. We reran all 40 stories under all six actor models with the sentence removed from the actor prompt, together with the adjacent sentence that has a similar effect. Everything else is unchanged, including the worldkeeper, the terminator, the budgets, and the judge. Each cell gives the mean deviation from canon under the original prompt, then after removal with its story-clustered 95% bootstrap interval, averaged over the five checkpoints of each replay.

ActorMoodPlot intensityTension progressRelationship warmth
GPT−0.59 → −0.51 [−0.74, −0.21]−1.38 → −1.05 [−1.45, −0.65]−1.59 → −1.55 [−1.92, −1.18]−0.81 → −0.73 [−1.16, −0.29]
Claude−0.73 → −0.69 [−0.94, −0.42]−1.08 → −1.05 [−1.48, −0.62]−1.30 → −1.45 [−1.82, −1.08]−0.99 → −1.07 [−1.57, −0.51]
Gemini−0.67 → −0.56 [−0.85, −0.27]−0.90 → −0.70 [−1.17, −0.23]−1.23 → −1.30 [−1.68, −0.92]−1.00 → −0.91 [−1.43, −0.39]
DeepSeek−0.59 → −0.51 [−0.77, −0.21]−1.04 → −0.90 [−1.28, −0.52]−1.24 → −1.25 [−1.62, −0.88]−0.82 → −0.89 [−1.38, −0.45]
Kimi−0.97 → −0.92 [−1.17, −0.64]−0.05 → +0.05 [−0.28, +0.38]−1.24 → −1.25 [−1.58, −0.90]−1.14 → −1.19 [−1.66, −0.73]
Qwen−0.90 → −0.81 [−1.01, −0.51]−0.54 → −0.42 [−0.82, +0.00]−1.38 → −1.40 [−1.71, −1.07]−1.25 → −1.15 [−1.63, −0.68]

On all four scores, every value under the original prompt lies inside the corresponding interval after removal. Tension progress, the largest deviation under every actor in the paper, stays 1.25 to 1.55 points below canon. Plot intensity stays below canon under five of the six actors, and Kimi sits at canon level as it did before. The flatness therefore does not come from this instruction. We report this rerun in Sec. 6.4 and App. I.


W2 / Q3: Characters Removed by the Presence Check

Thank you for the attention to this filtering step. The three-in-four result does not depend on the check, as the share is 73.6% with no filter against 74.3% with it. The check exists because the decision-level analysis asks whether a persona changes what a character does, and this question is only meaningful when the character has taken up the persona in the first place. Without the check, a persona that fails to change an action could simply be one that the model never adopted.

The concern is that the check keeps the characters whose personas sit close to the default, which would inflate the agreement we report. The table reports the share of decision points whose default action already lies inside the implied set, on the retained characters, the removed ones, and all characters with no filter, with character-clustered 95% bootstrap intervals.

ActorRetainedRemovedNo filterRemoved − Retained [95% CI]
GPT74.064.172.2−9.8 [−19.5, +0.5]
Claude73.068.971.8−4.1 [−13.3, +5.3]
Gemini80.879.780.5−1.1 [−11.4, +7.8]
DeepSeek73.569.072.3−4.5 [−12.8, +4.0]
Kimi71.769.771.2−2.0 [−9.6, +5.1]
Qwen73.574.974.0+1.4 [−5.6, +8.1]
Pooled74.371.873.6−2.5 [−6.1, +1.1]

The difference between removed and retained characters has an interval that includes zero pooled and for every actor. The removed characters are also not a fixed group that models find hard to play. Of the 63 characters removed under at least one actor, only 9 are removed under all six, and 95% of the removals involve the relations check or the knowledge audit, with 5% on traits and values alone. Most removals thus come from a slip on a fact or a relationship under some models, not from a character that is hard to play. Efficacy is likewise close, at 0.161 on removed and 0.150 on retained characters. The conclusion is therefore not limited to characters that are easy for the model to play. We have added this comparison to App. I and note it in Sec. 6.2.

AuthorsOpenReview ↗

Rebuttal to Reviewer XLVh (Part 2)

W3 / Q2: Pre-Freeze Memories and Private Knowledge

Thank you for this question. Yes, they are removed together, and the results are unchanged when they are kept. The persona card contains the character's pre-freeze memories and private knowledge because the character needs them as plot reference to play its role. They are part of the persona, so removing the persona at a decision point removes them as well. The factual memories that the removal keeps are a different item, the passages of the pre-freeze source text that the character retrieves at each action and its notes from the replay (App. B).

The removal leaves the confound the reviewer points to, that the character without its persona looks more cautious because it knows less and not because of the model's default. We tested this at all 2,824 decision points by rerunning the persona-removed step with the pre-freeze memories, private knowledge, and ability limits kept, and with only traits, values, relations, and motivation removed. Actor, temperature, and judge are unchanged. Columns follow Tab. 2, and brackets give 95% bootstrap intervals.

ActorInside (%), persona removedInside (%), knowledge keptEfficacy, persona removedEfficacy, knowledge keptShift, persona removedShift, knowledge kept
GPT74.074.2 [69.1, 79.0]0.170.15 [0.11, 0.20]+0.13+0.08 [+0.01, +0.14]
Claude73.073.3 [68.4, 77.6]0.160.14 [0.09, 0.18]+0.12+0.09 [+0.03, +0.17]
Gemini80.879.5 [74.8, 83.8]0.070.06 [0.03, 0.09]+0.03+0.03 [−0.06, +0.11]
DeepSeek73.573.1 [69.3, 78.0]0.180.19 [0.14, 0.23]+0.14+0.16 [+0.07, +0.25]
Kimi71.769.3 [65.0, 72.2]0.160.16 [0.12, 0.21]+0.14+0.14 [+0.07, +0.21]
Qwen73.573.7 [68.5, 77.4]0.160.17 [0.12, 0.22]+0.23+0.21 [+0.09, +0.32]

Shift, which measures how much stronger the action is with the persona than without it, stays positive for every actor, with an interval that excludes zero for five of the six. The share of defaults inside the implied set and the efficacy are also unchanged, with no actor moving by more than 2.4 points on the first or 0.02 on the second. The cautious default therefore comes from the model and not from missing knowledge. We report this rerun in App. I, note in Sec. 4.1 that the persona carries part of the pre-freeze plot, and have clarified in App. B what the removal keeps.


W4: Reliability of the LLM Judge

Thank you for raising this, which is a fair concern for any study that relies on an LLM judge. The judge sits closer to the human consensus than the annotators sit to one another, and the conclusions hold when human labels replace it. Three annotators labeled 300 decision points independently, without seeing any output of the judge or which condition an action came from. The agreement of around 0.5 needs context. It is an unweighted chance-corrected agreement over five labels, in which a difference of one step counts as full disagreement, and the sample is deliberately weighted toward hard cases, with half of the points being ones where the default leaves the implied set. The analysis itself uses the implied set, which admits an adjacent label and shows higher agreement.

On what the persona calls for, the judge agrees with the majority of the three annotators at .63 on the implied set, against .59 among the annotators, and on the action labels at .71 against .51 (Tab. 10). Recomputed from the human majority labels on the calibration sample, adherence is 0.73 against the judge's 0.69, and efficacy is 0.18 against the judge's 0.25, so the persona still changes a minority of decisions. The LLM judge is therefore a sound basis for the measurements in the paper. We have added this reading of the agreement statistics to App. H.


Thank you again for the thorough review. We hope these additions address your concerns.

Reviewer_TmeWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ANIMASK investigates how assigned personas and underlying language models jointly influence behavior in simulated story worlds. It combines story-level narrative scoring, repeated persona interviews, and decision-level persona removal. Experiments cover 40 stories, six actor models, and 3,846 decision points, with 2,824 passing the persona-presence criterion. The authors report limited narrative progress despite high interview-based persona scores, alongside 80–86% adherence and 7–18% operational efficacy. They interpret these findings as evidence that model defaults constrain behavior even when character identities remain recognizable.

Critical strengths
  1. A useful distinction between persona fidelity and behavioral influence.
    The framework goes beyond checking whether models can describe a character’s identity and values by examining whether those characteristics affect consequential decisions. Combining interviews, narrative outcomes, and local persona removal provides a valuable diagnostic perspective.

  2. Substantial experiments and transparent evaluation.
    The study includes multiple models, two languages/media, memorization screening, blinded action labeling, human calibration, story-clustered confidence intervals, and threshold sensitivity analyses. Detailed prompts and worked examples make the evaluation procedures inspectable.

Critical weaknesses
  1. The intervention does not adequately identify the persona’s causal contribution.
    Persona removal retains a history generated with the persona present, so agreement can reflect earlier persona influence rather than model defaults. Matching one of five broad action labels can also conceal meaningful behavioral differences.

  2. Shared prompting and simulation constraints confound the model-default explanation.
    The actor prompt explicitly highlights caution, waiting, and de-escalation while discouraging added drama (Appendix D.2). All actors also share the same environment machinery, without environment-side ablations. Replays terminate after two rounds without material change or upon exhausting their budget; 102 of 240 remain open and are still progressing when stopped (Table 12). These shared conditions could contribute to the observed convergence. Comparing truncated simulations with completed canonical narratives does not sufficiently isolate actor-model dispositions.

Questions for authors
  1. On a representative subset, do repeated samples with and without personas yield significantly different action-label distributions beyond a same-persona resampling baseline?
  2. Does the convergence persist after modifying the caution-oriented prompt, extending termination thresholds and budgets, or changing the worldkeeper? How often do currently open replays resolve when allowed to continue?
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer TmeW (Part 1)

We sincerely thank you for the careful and detailed review. Your questions led us to run four new experiments, and the findings of the paper hold under each. We address each point below. Section, table, and appendix numbers refer to the revised PDF.


W1: History Kept under Persona Removal

Thank you for the careful reading of the persona-removal design. The agreement between the default and the persona is already present where almost no history exists, so it is not carried over from a history generated with the persona. The history is kept so that both conditions face the same situation, and the comparison measures what the persona changes at one decision. Some influence of the persona may remain in the history, which is why we treat efficacy as a lower bound on the share of decisions the persona changed (App. A; Proposition 1 in App. G).

In the first round of a replay, the default already lies inside the implied set at 73.1% [64.3, 80.8] of 193 decision points, against 74.3% over all 2,824, and the share stays level across the three thirds of a replay. The persona-free replays of GPT, whose history contains no persona at any step, deviate from canon in the same direction on all four scores (App. I). This supports reading the no-persona action as the default of the model. We have added these shares to App. I.


W1 / Q1: Coarse Action Labels and a Resampling Baseline

Thank you for suggesting this baseline. We ran it, and the label distributions with and without the persona differ beyond the resampling baseline at medium-risk and high-risk points, which make up 271 of the 300 sampled points. We drew 300 of the 2,824 decision points, stratified by actor and language (50 per actor, 39 stories), and sampled each action five times with the persona and five times without it, for 3,000 judged actions. For each point we compute how often two draws receive different labels, within each condition and between the two. The excess is the between-condition rate minus the mean of the two within-condition rates. It is zero in expectation when the persona does not change the label distribution, or when the labels are too coarse to register a change.

Risk level (points)Within, with personaWithin, without personaBetween conditionsExcess [95% CI]p (permutation)
Low (29)0.1380.0900.108−0.006 [−0.019, 0.000]0.75
Medium (154)0.1880.1900.221+0.032 [+0.011, +0.055]<0.001
High (117)0.2350.1490.251+0.059 [+0.029, +0.097]<0.001

The excess is positive at medium-risk and high-risk points (p < 0.001). At the 29 low-risk points there is no excess. The excess is also small, which matches the finding of the paper that the persona changes a minority of decisions.

On granularity, the five labels code action strength, which is the quantity our claims concern. A shared label means the same action strength, not the same text (Sec. 3). Counting only the called label without its adjacent label changes adherence by at most 0.011 and efficacy by at most 0.002 (App. I), and the judge agrees with the majority of three annotators at .71 on these labels (Tab. 10). The label scale is thus fine enough for the claims we make. We have added this baseline to App. I.

AuthorsOpenReview ↗

Rebuttal to Reviewer TmeW (Part 2)

W2 / Q2: Caution-Oriented Actor Prompt

Thank you for singling out this instruction. The convergence persists without it across all 240 replays. Our study does not aim to continue stories. It uses the story as a setting to measure what the persona and the model each contribute to a character's actions. A model in a narrative setting may dramatize simply because it is in a story, so we added this instruction as a precaution. To test its effect, we removed "Never add drama for its own sake" and the related wording that hints at caution from the English and Chinese actor prompts and reran all 40 stories under all six actors, with everything else unchanged. Each cell gives the deviation from canon under the original prompt, then after removal with its story-clustered 95% bootstrap interval.

ActorMoodPlot intensityTension progressRelationship warmth
GPT−0.59 → −0.51 [−0.74, −0.21]−1.38 → −1.05 [−1.45, −0.65]−1.59 → −1.55 [−1.92, −1.18]−0.81 → −0.73 [−1.16, −0.29]
Claude−0.73 → −0.69 [−0.94, −0.42]−1.08 → −1.05 [−1.48, −0.62]−1.30 → −1.45 [−1.82, −1.08]−0.99 → −1.07 [−1.57, −0.51]
Gemini−0.67 → −0.56 [−0.85, −0.27]−0.90 → −0.70 [−1.17, −0.23]−1.23 → −1.30 [−1.68, −0.92]−1.00 → −0.91 [−1.43, −0.39]
DeepSeek−0.59 → −0.51 [−0.77, −0.21]−1.04 → −0.90 [−1.28, −0.52]−1.24 → −1.25 [−1.62, −0.88]−0.82 → −0.89 [−1.38, −0.45]
Kimi−0.97 → −0.92 [−1.17, −0.64]−0.05 → +0.05 [−0.28, +0.38]−1.24 → −1.25 [−1.58, −0.90]−1.14 → −1.19 [−1.66, −0.73]
Qwen−0.90 → −0.81 [−1.01, −0.51]−0.54 → −0.42 [−0.82, +0.00]−1.38 → −1.40 [−1.71, −1.07]−1.25 → −1.15 [−1.63, −0.68]

Every value under the original prompt lies inside the interval after removal. The shared direction is thus not explained by this wording (Sec. 6.4, App. I).


W2 / Q2: Shared Environment and Worldkeeper

Thank you for this question, which led us to run the environment swap that the paper had left open. The central pattern holds under a different environment model. The environment is identical across actors and conditions, so it cannot produce the differences between actors or between actions with and without a persona. To test whether it produces the shared direction, we replaced the model behind the worldkeeper and the archivist, GPT-5.6 Sol, with DeepSeek-V4-Pro and reran all 40 stories with GPT as the actor. The terminator, the prompts, and the budgets are unchanged.

ScoreOriginal environment [95% CI]Environment replaced [95% CI]
Mood−0.59 [−0.86, −0.32]−0.05 [−0.31, +0.23]
Plot intensity−1.38 [−1.79, −0.96]−1.84 [−2.31, −1.45]
Tension progress−1.59 [−1.96, −1.23]−1.52 [−1.91, −1.18]
Relationship warmth−0.81 [−1.28, −0.35]−0.28 [−0.73, +0.21]

Plot intensity and tension progress, the two largest deviations under GPT, remain below canon with intervals clear of zero, and tension progress is nearly unchanged. Mood and relationship warmth are more sensitive to the environment model and move toward canon. The central pattern, replays that stay quieter than the canon and leave its central tension less settled, therefore does not depend on the original environment model. We report this swap in Sec. 6.4 and App. I and have updated the corresponding limitation.


W2 / Q2: Termination, Budgets, and Open Replays

Thank you for these detailed questions. Giving the replays twice the room changes little. A standstill is the verdict of the terminator that the story has stopped moving, not a truncation, and 118 of the 240 replays end this way (Tab. 13). The trajectories also do not resemble canon trajectories cut short. Over the five checkpoints the canons gain 3.05 points on tension progress, and the replays gain 0.38 to 0.93 (Tab. 12).

To test this directly, we raised the standstill threshold from two to four consecutive rounds without material change and doubled the round budget to eighty rounds and the wall-clock limit to twelve hours. The engine keeps no resumable state, so we reran all 40 stories with GPT from the freeze point. Scores are deviations from canon with 95% intervals.

GPT, 40 storiesOriginal settingsLoosened settings
Standstill / resolved / open24 / 2 / 1425 / 3 / 12
Median rounds used1837
Mood−0.59 [−0.86, −0.32]−0.32 [−0.58, −0.05]
Plot intensity−1.38 [−1.79, −0.96]−1.23 [−1.66, −0.81]
Tension progress−1.59 [−1.96, −1.23]−1.55 [−1.95, −1.23]
Relationship warmth−0.81 [−1.28, −0.35]−0.52 [−0.99, −0.05]

The replays run twice as long, and the endings barely change. Open replays go from 14 to 12 and resolved ones from 2 to 3, and all four deviations remain below canon. The comparison with the canons therefore does not depend on where the replays stop (Sec. 6.4, App. I).


Thank you again for the detailed review. We hope these additions address your concerns.

AuthorsOpenReview ↗

Follow-up for Reviewer TmeW

Thank you again for your review. The discussion with Reviewer XLVh touched on three of the points you raised, and the revised PDF now contains further analysis on each. We summarize the additions here so that our earlier reply and the current PDF read consistently. Section, table, and appendix numbers refer to the revised PDF.

Resampling Baseline and the Size of the Persona's Effect

The baseline you suggested (Tab. 22) shows that the labels with and without the persona differ beyond resampling. To also give the size of the persona's effect net of sampling variation, we added a second analysis over all 2,824 decision points. If the persona had no effect, the draw with the persona and the draw without it would be interchangeable, so the case where the action lies inside the implied set and the default lies outside would be as frequent as the reverse case. The first case occurs at 14.8% of the points and the reverse at only 5.8%, a net gain in adherence of 9.0 points (95% interval [7.2, 10.8]). The gain is 8.5 to 12.1 points under five actors, with every interval clear of zero, and 1.7 points under Gemini, whose default is already the closest to its personas. Sampling variation cannot produce this gain. Sec. 6.3 now reports it alongside efficacy, and Tab. 24 in App. I gives the values for each actor.

Granularity of the Action Labels

A persona does not prescribe a unique action. A forceful course can take many forms, and repeated samples under the same persona already differ in their concrete actions. What a persona prescribes is how far the character goes at a choice, and this is what can be compared across what the persona calls for, the actual action, and the default action. The five labels code this degree, and the paper draws no conclusion about differences within a label. Two phrases in Sec. 6.3, "make the same decision" and "persona-induced behavioral change", now read the same action strength and a persona-induced change in action strength. We did not adopt a finer scale. Under five labels, two annotators assign exactly the same label at only 57% to 65% of the points on average, and no subdivision of the five labels can raise this share.

Prompt Wording

Our earlier reply reported a rerun that removed the caution-oriented wording from the actor prompt. Following a suggestion from Reviewer XLVh, we extended the removal to the whole engine, including the environment-side prompts and the instruction "Do not steer toward drama or resolution", and reran all 40 stories under all six actors. Tab. 17 now reports this stronger ablation in place of the actor-only rerun, so its values differ from the table in our earlier reply. The conclusion is the same. Of the 24 deviations, 23 remain negative, and tension progress stays below canon under all six actors, with every interval clear of zero. The wording has a small effect. Averaged over the six actors, less than a third of the original gap closes on tension progress and plot intensity, and mood and relationship warmth barely move.

Environment Model

Sec. 6.4 now states explicitly that the environment model affects the story-level scores, as Tab. 18 shows for mood and relationship warmth, and that this is why the environment is held fixed. The environment is identical across all actors and all conditions, and decisions with and without the persona are compared under the same environment and history, so the environment cannot produce that difference. Sec. 5.1 and Sec. 6.4 describe the story-level results as a tendency observed under a fixed environment.

We hope these additions are useful for your assessment.

Reviewer_gAWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ANIMASK, a simulation framework designed to investigate whether the behaviors of LLM-based role-playing agents are driven by assigned personas or the actor models' inherent behavioral tendencies. The framework constructs simulated story worlds from books and scripts, assigns character personas through prompt-based conditioning, and analyzes behavioral differences by comparing actions generated with and without persona information. The authors evaluate persona and model contributions at three levels: story-level tendencies, persona presence, and decision-level behavioral contributions.

Critical strengths

1. Novel evaluation framework for disentangling persona and model contributions.

The paper introduces a novel evaluation approach that goes beyond conventional persona fidelity assessments. By employing persona removal experiments and the proposed efficacy metric, the authors distinguish between the actor model's default behavioral tendencies and the behavioral changes induced by persona conditioning. This provides a useful perspective for understanding the actual influence of assigned personas on LLM behavior.

2. Comprehensive experimental evaluation across diverse stories and models.

The paper conducts extensive experiments involving 40 stories and 6 LLMs, analyzing 3,846 decision points. Furthermore, the evaluation considers multiple levels, including story-level tendencies, persona presence, and decision-level behavioral contributions. This multi-level experimental design provides comprehensive evidence supporting the paper's findings.

Critical weaknesses

1. Insufficient justification of the practical significance of disentangling persona and model contributions.

While the paper investigates whether character behaviors are driven by assigned personas or the actor model's default dispositions, the practical significance of this distinction remains insufficiently justified. If an LLM consistently generates behaviors that align with the assigned persona, it is unclear why distinguishing whether those behaviors originate from the persona or the model's inherent tendencies is necessary. Although the proposed efficacy metric provides additional information beyond persona adherence, the paper does not sufficiently demonstrate how this distinction leads to practical improvements in persona modeling or role-playing agents. The authors should further clarify the practical implications of this analysis and explain how the proposed measurements can guide the development or evaluation of more effective persona-conditioned models.

2. Limited generalizability due to the exclusive use of prompt-based persona conditioning.

The paper relies exclusively on prompt-based persona conditioning, where character profiles, motivations, and memories are provided through prompts. However, this is only one approach to persona conditioning, and its ability to influence model behavior may be limited compared to training-based approaches. The paper concludes that the actor model's default dispositions constrain the extent to which a persona can influence behavior. However, this finding may partly reflect the limitations of prompt-based persona conditioning rather than an inherent limitation of LLMs in adopting assigned personas. More sophisticated approaches, such as supervised fine-tuning or parameter-efficient fine-tuning (e.g., LoRA), may lead to different results. Evaluating at least one training-based persona conditioning method would help determine whether the observed behavioral constraints arise from the model's inherent dispositions or from the specific persona conditioning method employed. Without such comparisons, the generalizability of the conclusions across different persona conditioning techniques remains limited.

Questions for authors

1. Practical significance of distinguishing persona adherence and efficacy.

The paper demonstrates that persona adherence can remain high even when persona efficacy is relatively low. However, what practical benefits does distinguishing these two metrics provide for the development or evaluation of persona-conditioned agents? Could the authors clarify situations in which measuring persona efficacy would lead to different conclusions or decisions compared to evaluating persona adherence alone?

2. Impact of persona conditioning methods on the findings.

The experiments exclusively employ prompt-based persona conditioning. How can the authors determine whether the observed behavioral constraints arise from the actor models' inherent dispositions rather than the limitations of prompt-based conditioning? Is there any evidence suggesting that similar patterns would persist under training-based persona conditioning methods, such as supervised fine-tuning or LoRA? Clarifying this distinction would help assess the generalizability of the paper's main conclusions.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer gAWE

We sincerely thank you for the constructive review. Your questions led us to state the practical value of efficacy more plainly and to run the training-based experiment you suggested. We address each point below.


W1 / Q1: Practical Significance of Separating Adherence from Efficacy

Thank you for this thoughtful question. Adherence shows that behavior matches the persona, whereas efficacy shows whether the persona produced it. A developer who assigns a persona needs to know whether the character will do what the persona specifies when the model would otherwise do something else. Matching behavior cannot answer this. At three decision points in four the model's default already satisfies the persona, so most of the match belongs to the model and would be there without the persona.

For example, Claude and Gemini share an adherence of 0.83 (Tab. 2), yet the persona changes the action at 16% of decision points under Claude and 7% under Gemini. Adherence ranks them as equal, whereas an application that needs characters to depart from the default would choose between them.

For evaluation, adherence is nearly flat across the six actors, while the persona's win rate against the default ranges from 36% to 68%. Efficacy and this win rate should be reported with adherence, and test personas should call for actions the default would not take. For development, characters describe their personas accurately in interviews and still act as the default does. The difficulty lies less in representing the persona than in turning it into action, most of all where the action is stronger than the model would take on its own. In short, adherence alone overstates how much control a persona gives, and efficacy measures that control directly.

We have made this reading explicit in Sec. 6.3.


W2 / Q2: Training-Based Persona Conditioning

Thank you for this suggestion. We ran the experiment, and the result under LoRA is consistent with our conclusion. The paper studies prompt-based conditioning because this is how the systems it addresses assign personas, including agent societies (Park et al., 2023; Zhou et al., 2024b), story worlds (Ran et al., 2025), and role-play benchmarks (Ahn et al., 2024; Samuel et al., 2025). Training-based conditioning (Shao et al., 2023; Lu et al., 2024) needs role-play data and access to the weights, which API-only models do not offer.

We compared prompt-based and LoRA conditioning on Qwen3-4B over all 40 stories, with one replay per story and condition. Each character has a LoRA adapter (rank 16, two epochs) trained on 20–60 examples from the source text before the freeze point. In the replay the adapter receives no persona card, so the weights alone carry the persona. To obtain the default of a LoRA character, we train a second, neutral adapter on the same contexts, with responses that the base model gives without a persona card. Comparing the two adapters separates the persona from the effect of fine-tuning itself.

ConditioningDecision pointsAdherenceEfficacyInside (%)Wins (%)
Prompt5030.704 [0.636, 0.768]0.155 [0.123, 0.189]75.1 [70.9, 78.9]57.6 [48.7, 65.9]
LoRA5170.741 [0.685, 0.793]0.178 [0.143, 0.209]75.6 [71.8, 79.2]63.5 [54.0, 71.3]
Prompt, common characters3460.714 [0.643, 0.785]0.156 [0.119, 0.199]75.1 [70.2, 79.5]58.1 [47.8, 68.4]
LoRA, common characters4210.736 [0.667, 0.797]0.173 [0.135, 0.210]75.1 [70.5, 79.1]61.9 [51.5, 70.6]

Columns follow Tab. 2, and brackets are 95% story-clustered bootstrap intervals. The presence criterion retains 90 characters under prompting and 76 under LoRA, and the last two rows keep the 62 retained under both.

The two methods give the same picture. Under both, the default action already agrees with the persona at about three decision points in four, and where the two disagree, the persona decides the action about six times in ten. No measure differs between the methods beyond its interval, and the 62 common characters give the same result. A persona carried by the weights is therefore bounded by the model's default in the same way as a persona given in the prompt. Lu et al. (2024) report a related limit for knowledge under role-play fine-tuning, where the seed model confines role-specific knowledge. The LoRA result is thus consistent with the conclusion of the paper, which does not depend on prompt-based conditioning. We report this experiment in App. I.


Thank you again for your review. We hope these additions address your concerns.

Reference not in the paper. Lu, K., Yu, B., Zhou, C., and Zhou, J. Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment. ACL 2024.

Thank you for your detailed response and additional experiments. While some of my concerns have been addressed, I still have a few questions.

  1. Efficacy vs. Win Rate: Since efficacy is calculated over all decision points, it may be low simply because the model's default behavior already aligns with the persona. What additional value does efficacy provide beyond the win rate in disagreement cases?

  2. Practical Significance: Efficacy seems to measure behavioral change rather than behavioral control. The authors should demonstrate the practical benefits of higher efficacy, for example, in scenarios where personas change dynamically.

  3. Generalizability: The additional LoRA experiments are helpful, but evaluating another persona conditioning method would further strengthen the generalizability of the findings.

Given these remaining concerns, I have decided to maintain my original rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer XLVh's Follow-up (Part 1/2)

Thank you for the follow-up and for your willingness to reconsider the score. We address each point below, in two comments. Section, table, and appendix numbers refer to the revised PDF.

W1 / Q1: Restrictive Wording on the Environment Side

Following your suggestion, we ran a prompt ablation that covers the whole engine, and the replays remain flatter than their canons. It removes every instruction on the actor side and the environment side that could hold back conflict or resolution, in the prompts for character action and response, consequence adjudication (including "Do not steer toward drama or resolution"), minor-character responses, scene casting, and time skips. App. I lists each change. The ablation covers all six actors and all 40 stories, with one replay per story.

Each cell gives the deviation from canon (replay minus canon, averaged over the five checkpoints and the 40 stories) under the original configuration → after the ablation, with the 95% interval of the latter from a clustered bootstrap over stories.

ActorMoodPlot intensityTension progressRelationship warmth
GPT−0.59 → −0.66 [−0.98, −0.36]−1.38 → −1.10 [−1.56, −0.66]−1.59 → −1.18 [−1.58, −0.82]−0.81 → −0.63 [−1.15, −0.12]
Claude−0.73 → −0.50 [−0.80, −0.18]−1.08 → −0.72 [−1.13, −0.32]−1.30 → −1.39 [−1.69, −1.04]−0.99 → −1.07 [−1.51, −0.66]
Gemini−0.67 → −0.79 [−1.00, −0.59]−0.90 → −0.48 [−0.94, −0.05]−1.23 → −0.88 [−1.31, −0.47]−1.00 → −0.73 [−1.14, −0.35]
DeepSeek−0.59 → −0.56 [−0.85, −0.26]−1.04 → −1.12 [−1.49, −0.78]−1.24 → −0.69 [−1.11, −0.27]−0.82 → −0.94 [−1.51, −0.40]
Kimi−0.97 → −1.08 [−1.34, −0.79]−0.05 → +0.18 [−0.16, +0.49]−1.24 → −1.02 [−1.35, −0.70]−1.14 → −1.01 [−1.42, −0.55]
Qwen−0.90 → −0.61 [−0.90, −0.31]−0.54 → −0.19 [−0.62, +0.23]−1.38 → −1.45 [−1.97, −1.00]−1.25 → −1.20 [−1.53, −0.84]

If the flatness came from this wording, it should disappear once the wording is gone, or mostly so. It does not. Of the 24 deviations, 23 remain negative, and tension progress stays below canon under all six actors, with every interval clear of zero. The wording has a small effect. The gaps on tension progress and plot intensity narrow slightly after the removal. Averaged over the six actors, less than a third of the original gap closes, and mood and relationship warmth barely move. The wording was there to keep a model from dramatizing simply because it is in a story, and the ablation matches that purpose. It held back a small amount of added drama and is not the main reason the replays depart from their canons. The flatness therefore does not mainly come from the prompt wording. We report this ablation in Sec. 6.4 and App. I, where Tab. 17 now gives this ablation in place of the earlier rerun that removed the actor-side wording only.

W4: Reliability of the LLM Judge

Your first concern is that human–human agreement is only around 0.5, so agreeing with the human majority does not show that the judge is reliable. The Alternative Annotator Test (Calderon, Reichart, and Dror, ACL 2025) addresses exactly this concern. It treats no human label as ground truth. It leaves out one annotator at a time, takes the remaining annotators as the reference, and compares whether the judge or the left-out annotator agrees better with that reference. The judge passes when it is no worse than the human, within a margin ε, for at least half of the annotators. We ran the authors' public implementation on the decision-point labels of the three annotators, at the three values of ε that the authors recommend (details in App. H).

LabelPointsAdvantage probabilityWinning rate, ε = 0.10ε = 0.15ε = 0.20
Called label3000.842/32/33/3
Actual label3000.943/33/33/3
No-persona label1500.912/33/33/3

The judge passes for all three label types under all three values of ε. The agreement of around 0.5 is unweighted, so a difference of one step counts as full disagreement, and on average two annotators are within one step of each other at 81% to 84% of the points. The judge can therefore stand in for a human annotator in the measurements of the paper.

Your second concern is that efficacy differs between the two sets of labels. On the 150 calibration points, the human majority labels give 0.18 and the judge gives 0.25. The judge's value is indeed higher. The difference is 0.07, and on this sample no difference is detected (95% interval [−0.01, +0.14]). More importantly, both sets of labels support the same conclusion. Whether efficacy is 0.18 or 0.25, the persona changes the action strength at a minority of decision points, and adherence is also close under the two sets of labels (0.73 and 0.69). The conclusions of the paper therefore do not depend on which labels are used. The revision reports both sets of results and the test above in App. H.

AuthorsOpenReview ↗

Response to Reviewer XLVh's Follow-up (Part 2/2)

Efficacy and Sampling Noise

The effect of the persona can be separated from sampling variation, and the data of the main experiment provide a direct test. If the persona had no effect, the draw with the persona and the draw without it would be interchangeable. The case where the action lies inside the implied set and the default lies outside would then be as frequent as the reverse case, since sampling variation cancels across the two directions. Over all 2,824 decision points, the first case occurs at 14.8% and the reverse at only 5.8%, a difference of 9.0 points (95% interval [7.2, 10.8]). This difference is the gain in adherence that the persona brings, and sampling variation cannot produce it.

ActorAction inside, default outside (%)Action outside, default inside (%)Gain in adherence [95% CI]
GPT17.15.5+11.5 [+6.7, +17.0]
Claude15.96.2+9.7 [+5.2, +14.6]
Gemini7.05.3+1.7 [−1.4, +4.8]
DeepSeek18.05.9+12.1 [+7.2, +16.9]
Kimi15.36.8+8.5 [+4.5, +12.3]
Qwen15.84.8+10.9 [+7.1, +15.2]

The gain is 8.5 to 12.1 points under five actors, with every interval clear of zero. Gemini's default is already the closest to its personas, and its gain is 1.7 points with an interval that includes zero, which also shows that the gain is not inflated by sampling variation when the persona's effect is small. You are right that efficacy contains sampling variation, but it does not reach 15 points. The implied set holds a single label at 98% of the decision points, so efficacy is almost the same as the share of points where the action is inside and the default is outside (15.0% against 14.8%). Label changes go in two directions, and if the persona had no effect, this direction would account for at most half of the same-condition change rate, about 9 points. The excess in Tab. 22 measures the rate of label disagreement, which is not the size of the persona's effect. The effect of the persona is therefore real. The core measurement of the paper is valid, and the overall results and conclusions are unaffected. The revision states in Sec. 6.3 that efficacy contains sampling variation and reports the gain in adherence alongside efficacy.

Granularity of the Action Labels

The paper does not claim that the persona leaves behavior unchanged. It asks whether an abstract persona is realized as behavior that most people would accept as fitting that persona. This cannot be judged at the level of concrete actions, because a persona does not prescribe a unique action. Take Isolde Varne in Sec. 6.3, whose persona places her supremacy above any moral restraint. A forceful course can take many forms, and repeated samples under the same persona already differ in their concrete actions. What the persona prescribes is how far she goes at this choice. The persona calls for a forceful action, she falls back to a more cautious choice, and the model without her persona makes the same retreat. The five-label scale provides a common yardstick, at this level of degree, for what the persona calls for, the actual action, and the default action.

So you are right that actions carried out differently within one level receive the same label. This is the intent of the scale, and the paper draws no conclusion about differences within a level. Two phrases in Sec. 6.3, "make the same decision" and "persona-induced behavioral change", were not precise enough, and the revision changes them to the same action strength and a persona-induced change in action strength. On strength, the five labels do resolve the effect of the persona, with the net gain of 9.0 points above. We did not adopt a finer classification. Under five labels, two annotators assign exactly the same label at only 57% to 65% of the points on average, and no subdivision of the five labels can raise this share.

Role of the Environment Model

Thank you for pointing out the mismatch between Sec. 6.4 and Tab. 18. Tab. 18 shows that the environment model affects the story-level scores, and that sentence in Sec. 6.4 was too broad. The revision corrects it. This is also the reason the paper holds the environment fixed. The paper studies what the persona and the model each contribute to a character's behavior, not how much of the flatness comes from the actor and how much from the environment. The environment is identical across all actors and all conditions, and decisions with and without the persona are compared under the same environment and history, so the environment cannot produce this difference. We therefore agree that the flatness cannot be attributed entirely to the actor model, and the conclusions of the paper do not depend on that attribution. The revision describes the story-level results in Sec. 5.1 and Sec. 6.4 as a tendency observed under a fixed environment, and notes in App. A that the environment swap was run on one actor only.

Thank you again for the careful follow-up.

The follow-up addresses some of my concerns, but I still have one: the measurement itself is not reliable.

The Alternative Annotator Test only shows the judge is no worse than one human annotator, and the reference is still annotators who often disagree. If humans can't agree, "no worse than a human" doesn't make the measurement stable. The 81–84% within-one-step agreement doesn't help either, since the implied set is a single label at 98% of points.

Recent work shows the same problem. LLM judges agree with humans at near-random level on persona-driven dialogue (κ ≈ 0.11, humans 0.29; Simple Agents, Biased Judges, ACL 2026). When facts and commitments in a narrative are implicit, evaluators disagree even on basic questions (Can LLM Agents Stick to the Script?, ICML 2026). Breaking the judgment into explicit, checkable items makes evaluation much more reliable (Check Your Work, ACL 2026). The paper's own Table 10 shows this too: the knowledge audit, the only checkable item, has human–human agreement of 0.87, while all the holistic judgments are 0.29–0.59. Both levels of the paper's conclusions rest on these.

I understand this field is subjective, and I appreciate the effort. At minimum, the judgments should be broken into checkable items with acceptable human agreement. That is the basic requirement. What I really hope for is more: a truly stable way to measure narrative and role-play behavior. That would be a huge contribution to the field and would deserve a much higher score. A fun project only needs to be fun, but science needs a well-defined question and reliable measurement, or it stays on the surface. Until this is solved, I can't give the paper stronger support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.