acceptodds
Under review as a conference paper at ICLR 2027

How Far Do Persona Effects Generalize in Language Models?

Abstract

Persona prompts ask language models to answer as particular kinds of people. We test whether relationships learned from these effects predict responses to new questions and remain useful across models and prompts. Across 57 attributes, three behavioral domains, and seven pairs of open 7 to 9B checkpoints, persona effects can be predictable without being portable. Separate attribute and task gains improve prediction beyond shared scaling significantly in OLMo-3 and Qwen2.5, with the most robust evidence in OLMo-3. Across model transfers, borrowed gains with one amplitude underperform shared scaling in most directions; allowing two target parameters removes the significant losses but yields no significant benefit over target shared scaling. After rewording, refitting significantly outperforms reuse with one amplitude in all six tested pairs, while changes of examples or country context often preserve reuse value. In the tested prompt transfers, regularized updates outperform both reuse strategies in median at 64 target questions per attribute. A separate survey comparison finds that selecting the more responsive checkpoint can worsen human fit; responsiveness is confounded with training status, and temperature calibration largely removes this cost but not errors in group ordering. Within the tested gain representation, apparent transfer can come from target calibration; source relationships must add predictive value beyond calibration and regularization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
persona prompting, post-training, transfer, LLM evaluation, human simulation, distributional alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_E2wiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper assesses whether a model's answer with a given persona is consistent across different training checkpoints (e.g., base, IFT, post-train). Models are prompted using a set of attributes from a different paper and then the differences from the base model are compared using some paper-defined metrics. A held out analysis on OpinionQA suggests that persona-sensitive models do worse.

Critical strengths
  • Experiments used multiple prompts and multiple models (though some of these are more dated), both of which could support a more general finding. Given the broad interest in persona prompting, this scale is helpful for ensure results are not model or persona specific.
Critical weaknesses
  • The paper does not do enough to contextualize its approach with respect to related work. Persona prompting have several years of work now and the paper only lists a few examples in Section 7 and later in Appendix K. None of these discussions really helped me understand what new gap this paper was addressing or how the current experiments relate to what is already known. A good example of this is the rewording experiment in 4.2, which is an already-established general result from multiple papers (models are sensitive to changes to prompt wording). What new is being learned here?

  • I had a very hard time following the paper, despite knowing this subarea well. I struggled to answer some of the basic questions about what types of personas were used (e.g., what attributes), what exactly is being asked, and how some ground truth is measured. These answers are buried in the appendix, but are basic setup, so I would want them earlier. The writing style itself is also hard to follow. Sections 2.1 and 2.2 are particularly tough examples where new metrics are introduced but it's not clear why we need to use these metrics instead of those from prior work. As a result, they feel more like unnecessarily added complexity.

  • This is related to the writing, but I genuinely am still not sure what we learned from all these experiments. I understand that the paper is trying to model the relationship between model training and persona consistency/fidelity, but why would this matter in practice? Is the motivation that we should use models from an earlier training stage to get better prompting behavior? The paper presents a large number of analyses and robustness tests, but, in part due the presentation, it's surprisingly tough to tell what we've learned from all of the studies, or what the impact of the results should be on future work.

  • One aspect I struggled with was whether these differences are meaningful. The authors note that some of the differences are within the scale that switching from bf16 to fp32 changes the results (line 660). Further, scale of the log-odds in C.4 suggests that the differences are small in their effect. Given that the focus for prompting here is on persona prompting, I would have wanted to see more results like in Section 6 that assess accuracy with respect to known group variation.

Questions for authors
  • All of the persona results are pooled together, which is helpful for visual clarity. However, I'm curious whether you saw any category-level differences that were interesting.
Reviewer_nuwLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper tests how well the effects of adding persona statements to prompts generalize when simulating different groups’ perspectives. The researchers put much efforts on post-training persona effects and evaluating the effectiveness and boundary of predicting models' unseen behaviors from post-training results.

Critical strengths

This paper is a good attempt to test how well the effects of adding persona statements to prompts generalize when simulating different groups’ perspectives.

Critical weaknesses

The authors assess persona effects by measuring the changes in the log odds of answers after adding persona statements, and claim that human fidelity should be validated as a separate dimension. This evaluation is too coarse to convince me that their measurement actually captures whether adding persona effectively changes models' perspectives. A strong alternative explanation is that the observed changes merely reflect differences in prompt wording.

In addition, I doubt their hypothesis that persona effects can be trained and generalized across questions, even within the same domain. In human studies, there is no guarantee that, at the aggregated level, participants with specific attributes will response related questions in the same direction, not to mention a shared effect size. This might be because related questions measures distinct concepts, or simply because of differences in question framing.

Overall, to support their claims, this paper needs a more fine-grained evaluation and clearer scope of generalization, as well as very careful data preprocessing. Addressing these concerns could either significantly reduce the contribution or require substantial additional work.

Questions for authors

In section 2.3, why testing reuse? I think this needs more substantial motivation. Generally, I would not expect post-training effects to propagate across models. In many RL papers, researchers repeat their methods across models and tasks, but only compare the qualitative findings rather than predicting training effects from results of other models.

Reviewer_ZXi7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether persona effects in language models—measured as the change in answer log-odds when a single persona attribute changes. The authors introduce a "gain" that rescales a Base checkpoint's attribute effect into the corresponding post-trained effect, and compare several predictors.

Critical strengths

The paper asks a question people actually care about: if a model changes its answers when you tell it "you are a retiree in India," can we learn something from that change that carries over to new questions? The useful move is that it splits this into two separate questions that are easy to conflate: is the pattern predictable, and does it travel? It turns out the answer is yes to the first and largely no to the second, and that distinction is genuinely clarifying.

Critical weaknesses

The practical significance of the effects is not established. The gains are reproducible but small in absolute terms (about 4–8% of squared error in the strongest pair), and the 1-point Q² reference is explicitly not a validated utility threshold.

For a paper whose stated motivation is persona-based survey simulation, the absence of any downstream demonstration—e.g., whether the gains change subgroup rankings or effect-size estimates in a simulation pipeline—is a material gap. Without it, the reader cannot tell whether the predictable structure matters for the applications the paper motivates.

Questions for authors

Can you report whether the fitted gains change any downstream simulation quantity? are there attributes where post-training creates an effect the Base lacks, and how would the framework handle them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.