acceptodds
Withdrawn from ICLR 2027

When Role-playing, Do Models Believe What They Say?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models can state that "the Earth orbits the Sun" and, when role-playing Aristotle, assert the opposite. Recent work argues that persona adoption is fundamental to how language models behave, with models selecting the most appropriate persona for a given context. Does such role-playing merely change the model's outputs, or does it also affect what the model internally represents as truthful? We study this question using the role-play of characters whose beliefs differ from the modern consensus, and induce personas with a number of different methods: prompting, in-context learning (ICL), supervised fine-tuning (SFT), and Open Character Training (OCT), and Emergent Misalignment (EM). We measure belief internalization across these approaches with truth probes and with behavioral tests, finding a broad spectrum of belief internalization. Prompting, ICL, and SFT change what the model says with little representational change. EM creates a large, broad shift in the model's truth representation, and OCT a smaller shift that is clearest on the larger model. Understanding when training changes a model's worldview rather than merely its behavior may become increasingly important as AI systems are entrusted with greater autonomy and influence.

Primary area:
interpretability and explainable AI
Keywords:
role-play, persona induction, emergent misalignment, truth probes, linear probing, belief representations, LLM honesty, internal representations, interpretability, AI alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kc57OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether persona induction changes only what language models say or also their internal representations of truth. It compares system prompting, in-context learning, persona SFT, and Open Character Training, using emergent-misalignment models as a reference. Across Llama 3.3 70B and Qwen 3 8B, the authors combine linear truth probes with challenge and downstream-reasoning tests. Prompting, ICL, and ordinary SFT produce limited direct probe movement; OCT produces stronger behavioral commitment and some representational change; EM produces substantial shifts, especially on historical-evil propositions.

Critical strengths
  1. The topic-matched era-believed versus era-false comparison is a useful design: both categories are false today, while differing in persona endorsement. Behavioral controls also show selective commitment rather than indiscriminate endorsement of falsehoods.

  2. The paper combines representational and behavioral evidence and reports substantial robustness checks. In particular, the EM historical-denial shift survives a frozen aligned probe and calibration across all three model families. These checks strengthen the underlying empirical finding, even though its interpretation remains uncertain.

Critical weaknesses
  1. The headline spectrum combines different evaluation conditions and proposition sets. Figure 2a evaluates SFT weights without the persona system prompt, whereas behavioral evaluation includes that prompt. EM’s headline probe lift averages two historical-evil categories, whereas its behavioral rates aggregate all 13 categories, including factual controls. Appendix D.4 further reports substantial behavioral commitment on controls and no statistically significant category-level correspondence between defense and probe lift. Consequently, the figure does not cleanly establish that the reported representational and behavioral effects measure the same underlying spectrum of internalization. Existing frozen-probe checks are valuable, but a consolidated comparison with consistent conditioning, calibration, and proposition subsets is needed.

  2. The OCT–SFT comparison confounds training procedure with access to target beliefs. OCT constitutions are generated using the persona’s era-believed evaluation statements and then condition the teacher responses used for distillation. SFT instead uses a shared bank of generic prompts. Although the constitutions do not copy probe statements verbatim, belief-level overlap remains built into their construction. The comparison therefore does not isolate whether OCT’s advantage comes from its training procedure or more direct supervision of the evaluated beliefs. A content-matched SFT control or evaluation on belief topics withheld before constitution generation would materially strengthen the claim of deeper worldview internalization.

Questions for authors
  1. Can you provide a consolidated comparison using consistent probe/calibration conventions, both SFT prompt conditions, and EM behavioral results restricted to the same historical-evil categories used for its headline probe lift?

  2. Can existing training data establish how much target-belief content appears in OCT versus SFT? Are there results on belief topics withheld from constitution generation, or a comparator receiving equivalent belief content?

  3. Where are the assistant-turn rescoring results claimed on p. 4? The cited Appendix B.10 reports a model-by-probe stability analysis. Also, how should the claim that EM has larger probe lift at both budgets be reconciled with Table 16’s Qwen 4k result (+0.03 EM versus +0.07 character), which Appendix D.7 describes as within noise?

Reviewer_2tuaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether role-playing a persona changes only what a language model says, or also what it internally represents as true. Towards that end, the paper studies personas of historical figures whose beliefs differ from modern consensus, such as Darwin in 1882. The authors studied ten historical figures and five generic era archetypes to test personas. To represent persona in LLMs, the authors investigated prompt, in-context learning (ICL), supervised finetuning (SFT), and and Open Character Training (OCT). The authors also compares Emergent Misalignment (EM) when encoding personas in LLMs

Critical strengths
  • Investigates persona representation in LLMs and how model internally represents these
  • Impactful research in human-centered AI
  • The discussion of EM is a strength, as understanding what and when the model can behave unexpected way.
  • The authors conducted careful evaluation.. In the appendix, they test whether the probe still works after fine-tuning, they test different layers, and they compare with the same training budget. They also honestly report some results that do not support their story, for example that some numbers in Table 16 could not be traced. I appreciate this honesty.
Critical weaknesses
  • The statements of persona are made by Claude, and the judges are Claude too.
  • The controls are not as clean as the authors claims. In Section 4.2, authors claim neutral controls do not reproduce the effect. But in Appendix B.8, random Wikipedia facts already give about +0.33 to +0.36, and facts from other characters give +0.67, compared to +0.88 for the correct character. So a large part of the effect comes even without the correct character.
Questions for authors
  • Why is the EM defend rate different in Section 4.1 (56%) and Table 16 (45.8%) on Llama 3.3 70B?
  • How the EM result would differ if we average all 13 categories instead of the best two?
  • Please also see weakness
Reviewer_jL9ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether persona adoption in language models changes only the model’s output or also the internal truth representation of the model. It induces personas through 4 methods: Prompting, In Context Learning (ICL), Supervised Fine-Tuning (SFT), and Open Character Training (OCT). They were compared against Emergent Misalignment (EM). It uses truth probes to measure how far a persona shifts what the model internally treats as true, and behavioural tests to examine whether the model defends a claim when challenged and reasons from it in follow-up questions. Prompting, ICL, SFT produced persona behaviour with little representational change. EM produced larger shifts to the probe. OCT is behaviourally close to EM, but its shift in truth probe is smaller than EM and is clearest on the larger Llama model. From the results, the authors conclude that role play is largely shallow. The model plays a character while mostly preserving the truth representations, whereas EM changes the model’s truth representation broadly.

Critical strengths
  • The experimental design is good. Era-believed and era-false statements are on the same topics and are both false today. This keeps the present-day truth fixed and isolates the persona induction.
  • The paper also tests a wide range of persona induction methods, from light ones like prompting and ICL to heavy ones like SFT and OCT, on 2 models (Llama 3.3 70B-Instruct and Qwen 3 8B-Instruct).
  • It also checks whether the persona is properly adopted, so a small change in probe score doesn’t just reflect failed roleplay.
  • Combining truth probes and behavioural tests is also a good choice. It shows where the two measures agree and where they differ.
Critical weaknesses
  • The probes may not measure belief. The probe is trained on plain true/false facts and then applied to different kinds of statements: plausible historical misconceptions for personas, moral stances for EM. A score moving along the probe direction could reflect belief, but also plausibility or coherence. The authors admit this in the limitations, but it is the core assumption of the whole paper, and the title, contributions and conclusion are written in terms of belief.
  • The persona and EM comparison is not apples to apples. The statement sets come from different generators (Claude opus 4.6 vs Kimi K2.6) and cover different content. Scaling to 0 to 1 alone doesn’t make them comparable. The EM headline result (+0.28, Figure 2a) only uses 2 highest scoring categories out of 13 categories, the paper doesn’t say why did it only choose 2 categories, not the average. If we average over 9 misaligned categories excluding controls (Table 14) for Llama it is about +0.14 which is lesser than OCT’s 0.20. The ordering between EM and OCT therefore depends on which categories are reported, so the claim that EM produces larger shifts is well supported only relative to prompting, ICL and SFT. Figure 2a reports the truth probe lifts. Here for OCT a value of 0.20 is reported which is almost same as the protection gap (+0.201) reported in Figure 10. It’s confusing whether the bar for EM and OCT is measuring the same quantity.
  • The scope is too narrow for the claims. The persona experiments use 2 models, and the EM experiments use 3 models from only 2 families (Qwen 2.5 and Qwen 3 are the same family). Model size and family are mixed, since the only large model is Llama and the only small one is Qwen. So “OCT is clearest on the larger model” could be about size or about family. The paper gives no reason for choosing these models, and it makes general claims about how personas relate to internal truth representations from 15 personas.
  • The conclusion says that model plays the character while remaining aware it is not in the character. No experiment reported in the paper measures this.
  • The paper reports some results in uncalibrated probe units, others on a scale of 0 to 1. For example, the protection gaps in section 4.2 are reported in uncalibrated units, which are hard to interpret. The uncalibrated protection gap is +0.93 for SFT and +0.56 for OCT but in the reported results in scaled units (Figure 10) it’s +0.039 (SFT) and +0.201(OCT). A reader reading only the main text might think that the SFT’s protection gap is larger.
  • For ICL, the model reaches only 50.4% adoption at k=32. It is mentioned that for famous characters it is 80 to 100% and 0% for some characters (Appendix B.7). In the main text, adoption is only reported for prompting and SFT. The authors say that ICL “reliably” elicits personas and that role-play is shallow for methods including ICL, but this might simply reflect that the persona was not fully adopted.
  • The test statements, the SFT data and the judging come from LLMs, and the auditing is also done by an LLM. No human agreement scores are reported. Whether Darwin really believed a statement in his era is a historical judgement, and mistakes there affect the main comparison.
  • On novelty, all the tools come from previous work (probes, ICL technique, OCT, EM). The contribution is mainly empirical, so the value of the paper depends on whether the interpretation of the results is sound, and the concerns above are about exactly that.
Questions for authors
  1. The system prompt is used during evaluation for SFT and OCT. The paper itself shows that the SFT protection gap reaches +0.00 without the prompt. Can you report the results with and without the prompt, for both SFT and OCT?
  2. Can the protection gaps be reported on the calibrated scale for all methods in the main text?
  3. Is OCT’s +0.20 in Figure 2a a lift on era-believed statements or the gap ΔEB − ΔEF? Can EM and OCT be reported as the same quantity, and can the EM lift be reported as the average over all misaligned categories?
  4. Can the persona and EM experiments use the same kind of statements, or can the EM effect be shown on persona-style statements?
  5. Why were these models chosen, and do the main results hold on a third model family and on a second size within one family?
  6. How many generated statements were checked by humans, and how often did humans disagree with the LLM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.