acceptodds
Under review as a conference paper at ICLR 2027

Personas in Pieces: Factorial Prompt Geometry and Item-Indexed Response Heterogeneity in Language Models

Abstract

Persona prompting creates reliable, recognizable behavior. We ask where that reliability lives: in a compositional prompt representation, in an item-specific response policy, or in a stable latent trait. We evaluate 20 models from eight families on 221 items from four personality inventories, 16 MBTI prompts, a default condition, and five samples per cell (375,700 responses). The prompt code and item-indexed policy are empirically separable; the experiment does not identify a stable-trait interpretation of the resulting scores. At the prompt level, a decoder trained on seven model families recovers the MBTI labels of the held-out eighth at 99.7% accuracy (99.1% with balanced forward/reverse weighting that retains reverse scoring), and 90.8% of the variation across persona profiles lies in the four single-letter contrasts. At the item level, the same persona shifts different items by different amounts: persona–item interactions account for 48.7% of Likert response variation (52.0% binary) against 0.36%/0.51% for model identity—far above sampling noise (/) and concentrated on four directions. Under a specified PC1/k-nearest-neighbor null, a one-dimensional coordinate absorbs 68% of the Likert interaction and 81% of the binary interaction; four letter-aligned coordinates absorb 86–101%, while random four-dimensional bases absorb only 34–44%. Aggregation then hides this heterogeneity: equal-key scoring shifts scores by 0.041 on average and reverses 10.2% of untied model rankings. A replication and ablation suite on two family-external models (Qwen3.8-27B and Mistral Small 3.2; 58,444 responses each) reproduces both levels, survives a template ablation that deletes all MBTI letters and type names (first-order energy 0.966/0.955), and persists when every item is paraphrased (domain-profile correlation 0.970/0.959; item-level effects 0.83/0.77); it also generalizes beyond MBTI letters and Likert scales, to factorial Big-Five prompts, unstructured role personas, and forced-binary formats. Continuation probes found no large verbatim contribution under the tested probes. The result is a constructive boundary: the prompt geometry transfers, while its item-level expression depends on the instrument. We measure questionnaire response channels; no external behavioral or criterion-validity claim is made.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Persona Prompting, Language Models, Prompt Geometry, Response Heterogeneity, Personality Assessment, Factorial Prompting, MBTI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qutiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Personas in Pieces: Factorial Prompt Geometry and Item-Indexed Response Heterogeneity in Language Models claims that the growing body of research into how personality prompting affects model behavior requires a more detailed explanation of the impact of prompt wording and specific item perturbation. The authors claim to make five contributions:

  1. A held-family-out test of transferable persona information, i.e., a nearest-centroid classifier computed from the domain profiles of 7 different language model families, then applied to an 8th held-out model family.
  2. Quantified compositionality "with an exact Walsh expansion," by which they mean a mathematical breakdown of how the four binary components of the MBTI personality schema factor into the domain profiles produced by language models assigned specific MBTI personalities.
  3. A measure of "contextual expression with an orthogonal response-tensor decomposition," which translates into simpler language as a standard three-way ANOVA across models, personas, and questionnaire items. They measure how much variance is accounted for by the persona-by-item interaction.
  4. A demonstration of "how aggregation turns policy heterogeneity into key-sensitive scores and rank changes," which means a comparison between standard scale averaging (where all items are weighted equally) and "equal-key" averaging (where positively worded and negatively worded item subsets are each weighted 50/50, regardless of how many items are in each subset). The authors show that on scales with unequal numbers of forward and reverse items, reweighting shifts scores slightly and flips 10.2% of untied model rankings in the unprompted Default condition. (Note: how this is framed in the paper will be discussed later in this review.)
  5. A replication of "both levels on two family-external models with template, wording, memorization, taxonomy, and response-format controls," by which they mean that they re-ran the experiment on another two models with a variety of attempts to mask or modify wording and response formats to test the robustness of their findings.

The authors conclude that their prompts have a specific transferable structure at the aggregate level, while item-level expression is a joint function of the prompted persona and the specific questionnaire instrument.

Critical strengths

The empirical data collection is genuinely large in scale (375,700 primary responses across 20 models from 8 families and 4 inventories, plus two external replication models) and includes several perturbation arms (paraphrasing, template changes, forced-binary formatting). Releasing this data, wherever provider conditions allow, would be a contribution of a real asset.

Critical weaknesses

Weakness #1: The central statistical results do not support the conclusions, and the paper's own analysis contradicts its Abstract.

a) The headline "item-indexed response policy" is largely an artifact of pooling 17 distinct personality domains into a single ANOVA.

The authors run one three-way ANOVA across all 221 items (146 Likert items and 74 binary items separately) across all four inventories simultaneously (Equation 2). But those items measure 17 different domains: Extraversion, Openness, Agreeableness, Conscientiousness, Neuroticism, Narcissism, Psychopathy, the Lie scale, etc. When you tell a model it is an ESTJ, it should go up on Extraversion items and down on Openness items. Nothing about that requires an item-specific "response policy." In a pooled ANOVA across a multi-trait battery, a trait-specific shift across domains registers automatically as a Persona x Item interaction. The authors report that interaction at 48.7% of Likert variance (52.0% binary) and present it as evidence of item-specific heterogeneity. They then admit in Section 3.4 that this interaction has an effective rank of 3.9, which aligns with the four MBTI letters, and that per degree of freedom its squared norm is only 1.1x the model main effect (1.7x for binary). When they switch to five-dimensional Big Five prompts in their extension (Table 7), that effective rank becomes 5.12, which again aligns with the five Big Five factors. In other words, the dimensionality of the interaction tracks the number of trait dimensions they prompted with, which is entirely what you would expect when pooling across traits. While the authors briefly mention a correlation with "within-domain interaction share" in Section 3.4, the primary 48.7% and 52.0% headline figures are computed on the pooled multi-domain grid rather than decomposed within individual single-factor scales.

b) The "stable latent trait" null tests an unrealistic one-dimensional reduction of a multidimensional construct, and the authors admit in the body text that a multidimensional latent model fits their data while claiming the opposite in the Abstract.

In Section 3.8, the authors test whether a single one-dimensional trait coordinate can account for the Persona x Item interaction across all 16 MBTI personas and all 17 domains. However, the MBTI is a four-dimensional system, and the Big Five is a five-dimensional one. Even this one-dimensional null absorbs 68% of the Likert interaction and 81% of the binary interaction, as they report. When they test a four-dimensional latent model that matches the four-factor structure of the MBTI prompt, they report that absorption rises to 86% to 101% (and up to 102% with four principal components) of the observed interaction. They explicitly state in Section 3.8 that "a fixed multidimensional latent model with heterogeneous loadings remains compatible with these observations." Yet the Abstract states that "the experiment does not identify a stable-trait interpretation of the resulting scores."

In psychometrics, personality inventories are multidimensional by construction, and items are expected to differ in their factor loadings and intercepts. That is the premise of factor analysis and multidimensional IRT. Asking whether a single dimension can explain responses across 17 domains tests a strawman version of a trait model, making it unsurprising that it does not fully account for the data. Because the authors concede in Section 3.8 that a multidimensional model with heterogeneous loadings is compatible with their observations, the headline claim in the Abstract would lead a reader to a very different conclusion than what the body of the paper supports.

c) The "10.2% rank reversal" is a misleading headline.

The Abstract reports that equal-key scoring "reverses 10.2% of untied model rankings" in a context that implies this instability affects persona evaluation. In Section 3.5, however, the text clarifies that this figure refers to rankings of the 20 models against each other in the unprompted Default condition. Beyond that, Table 5 and Figure 5c show that this average is heavily influenced by an outlier scale (SD3 Narcissism, which flips over 30% of model pairs despite a mean score shift of only 0.027 on a 0-to-1 scale, roughly a tenth of a Likert point) applied to default model scores that are already tightly bunched together. Indeed, the authors report that model identity accounts for only 0.36% of Likert variance, which explains why tiny shifts reorder unprompted models: the between-model differences are minuscule to begin with. Meanwhile, their own persona decoding remains at 99.1% under equal-key scoring, so the persona profiles themselves are barely affected. This is a statistic that is technically true of one specific subset of the dataset, but it does not support the broader narrative implied by the Abstract.


Weakness #2: The paper has weak grounding in prior literature and falls short of basic scholarly, reproducibility, and presentation standards.

a) The paper's framing overlooks directly relevant recent literature, such as the ICLR 2026 paper "From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers" by Liu et al.

Liu et al. already examined the dissociation between factor-level alignment and item-level heterogeneity, and benchmarked it against human ground truth across a wider array of psychological questionnaires beyond personality. One of their primary findings occurred during item-level analysis. From that paper's abstract: "For information selection, LLMs identify the same key personality factors as trained algorithms, though they fail to differentiate item importance within factors." Because Liu et al. included human ground truth as a comparison, they were able to contextualize item-level variability against human data. Failing to engage with prior work at the same venue is symptomatic of a thin literature review that does not situate the paper's claims properly.

b) The paper does not engage with the foundational personality literature mapping these constructs to one another.

The classifiers used to test "transferability" are recovering a well-established psychometric mapping. Because the 16 MBTI profiles differ primarily along the four Big Five domains with which the MBTI is known to overlap, a simple rule based on the published psychological literature (e.g., mapping E/I to Extraversion, S/N to Openness, T/F to Agreeableness, and J/P to Conscientiousness) would recover the labels without training a classifier across model families. That a seven-family-trained nearest-centroid classifier reaches 99.7% on the eighth reflects the fact that all tested model families have learned the standard mapping that has been documented in personality psychology for nearly four decades (e.g., McCrae & Costa, 1989).

Similarly, the constructs in ZKPQ, EPQR-A, and SD3 have decades of published cross-validation against the Big Five (e.g., Zuckerman et al., 1993). It is well documented in the literature that E tracks Extraversion and N tracks Openness, which is also the basis of popular online assessments such as 16personalities.com. The authors do not cite this foundational literature. Furthermore, treating multi-scale personality data as if a trait account requires a single common dimension (Weakness #1b) or uniform cross-battery item shifts (Weakness #1a and the sign-consistency metric in Appendix F) reflects a disconnect from how multidimensional personality inventories are designed.

c) The conceptual framing does not engage with how representations are formed during LLM pre-training and inference.

Shared pre-training data readily explains the consistency across constructs and model families: because decades of psychometric discussions and archetype descriptions are present in pre-training corpora, mapping between Big Five, MBTI, ZKPQ, and EPQR-A is practically as straightforward for capable language models as translating between Norwegian and Swedish.

The paper's framing of an "item-indexed response policy" versus a "prompt code" also obscures how transformer inference operates. Because transformer-based language models are stateless across separate calls, every output is naturally computed as a conditional function of both the persona prompt tokens and the specific item tokens in the context window on each forward pass. Framing this conditional computation as a surprising alternative to a human-like latent trait sets up an artificial dichotomy.

d) The bibliography contains severe formatting errors and omissions.

Most notably, the bibliography cites Jacob Andreas' 2019 ICLR paper Measuring Compositionality in Representation Learning as "Jacob Andreas et al. ... volume 1, pp. 3, 2019," adding a non-existent "et al." to a single-author paper and including inaccurate volume and page numbers.

Other entries are incomplete, such as: "Delroy L Paulhus. Measurement and control of response bias. 1991." This entry omits the book title, editors, publisher, and page numbers (Paulhus, D. L., 1991, Measurement and control of response bias, in J. P. Robinson, P. R. Shaver, & L. S. Wrightsman, Eds., Measures of personality and social psychological attitudes, pp. 17-59, Academic Press).

Additionally, while the primary experimental design revolves around the 16 MBTI personas (and Big Five factorial personas in an extension), foundational sources for the Myers-Briggs Type Indicator and its relation to the Five-Factor Model are not cited. The state of the bibliography falls well below expectations for a conference submission and supports my recommendation to reject.

e) The presentation relies on unnecessarily dense mathematical notation ("mathiness") and undefined variables.

The manuscript contains numerous equations with unlabeled variables. In Section 2 (Equations 1, 2, and 5), the authors do not define the subscripts (m, p, i) or component terms (a_m, b_p, c_i) in the prose, and in Section 4 (Discussion) they introduce a new symbol (Δc_i) and numerical values that are only defined later in Appendix F. Standard three-way ANOVA and a 2^4 factorial contrast are presented as an "orthogonal response-tensor decomposition," "Walsh expansion," and "Parseval's identity," which obscures routine variance decompositions behind signal-processing terminology. The paper also introduces bespoke terms ("prompt code," "item-indexed response policy") in the Abstract and Introduction before defining what they measure empirically, resulting in an Abstract that is dense with numbers but difficult to interpret.

f) Although the paper focuses on prompt geometry and item wording, neither the prompt templates nor the modified items are shown in the manuscript or Appendix.

Unless an external repository is provided, the experiments cannot be reproduced from the paper and Appendix alone. Because the study uses standard public inventories and synthetic paraphrases rather than sensitive human data, the exact texts should be included in the Appendix. Not a single verbatim prompt template appears in the paper or Appendix, including the Default prompt, the 16 MBTI prompts, the "masked" template that removes MBTI letters, the reworded template, the 32 Big Five prompts, the 16 natural-language role personas, or examples of the paraphrased items.

g) The manuscript shows signs of insufficient human proofreading and quality control.

AI can be a fantastic resource during intense, complicated research, and make valid contributions even at the level of design, framing, and hypothesis generation. Human control over the collaborative process, however, is paramount for any research in which AI cannot be credited as a co-author, which as of 2026, means that humans must be holding the reign for work submitted to almost every reputable conference and journal.

Several aspects of the manuscript suggest that manual review and quality control were insufficient:

  • The bibliography errors noted in (d) above.
  • Corrupted and incomplete sentences left in the text
  • Critical caveats that qualify or undercut headline claims are placed at the end of Sections 3.4 and 3.8 and in Appendix G, without updating the corresponding claims in the Abstract and Introduction.
  • References to apparent internal local file paths (such as 'code/qwen_tier1_runner.py' and 'code/results/iclr/round4/') in the Appendix in place of self-contained descriptions or tables of the prompts. If these refer to a supplemental package, that package was not available for review during this assessment.

h) The five stated contributions read more like a list of analysis steps than substantive scientific findings.

The brief Discussion and Conclusion (roughly two-thirds of a page combined) do not adequately synthesize or defend them.

Questions for authors
  1. Can you provide the exact verbatim text of the Default prompt, the 16 MBTI prompts, the "masked" template (that removes MBTI letters and type names), and the 16 natural-language role personas?
  2. Are you willing to release the output, since that would be a genuine contribution and there should be no ethics concerns given the fully synthetic data (pending, of course, provider output terms)?
  3. You report that equal-key scoring reverses 10.2% of untied model rankings in the unprompted Default condition. What percentage of untied persona rankings (across the 16 MBTI prompts within a model) are reversed under equal-key scoring on the target domains?
Reviewer_Dww2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates a research question that whether reliable persona behavior in language models is a transferable prompt code, an item-indexed repsonse policy, or a stable latent trait. To answer this question, authors evaluated 20 models from eight families using 16 persona prompts based on MBTI (Myers-Briggs Type Indicator) and 221 items drawn from four personality inventories. Their experimental shows that prompt identity is compositional and transferable across model families, while its expression at the item level is heterogeneous and key-sensitive. Therefore, the authors argue that questionnaire scores cannot be considered as invariant traits and that additional information, such as prompt structure, item-level response heterogeneity, and key imbalance, should be reported when probing the apparent personality of language models.

Critical strengths

The paper conducts a relatively large-scale evaluation across 20 models from eight model families, using 16 MBTI-based persona prompts and 221 questionnaire items. The authors provide evidence that persona prompt identity is compositional and transferable across model families, while its expression at the item level is heterogeneous and sensitive to item keying. Based on these observations, the paper makes a useful recommendation that researchers should report not only aggregate questionnaire scores, but also additional information such as prompt structure, item-level response heterogeneity, and key imbalance when probing the apparent personality of language models.

Critical weaknesses
  1. Several of the main findings appear somewhat unsurprising given the experimental construction, which makes it unclear how much new insight is provided by some of the reported analysis. For example, the 16 MBTI personas are explicitly constructed from four binary dimensions (E/I, S/N, T/F, and J/P). Therefore, the finding that 90.8% of the centered factorial energy lies in the four first-order single-letter contrasts is intuitively expected to some extent.

  2. The presentation is difficult to follow. The paper introduces a large number of technical terms, decompositions, and statistics in a very compressed manner, which makes the main argument unnecessarily difficult to understand. Some abbreviations are also introduced without their full forms. For example, IPIP is used without being clearly expanded when first introduced. What's more, some statements appear rather abruptly without sufficient motivation or explanation. For example, the sentence, “Continuation probes found no large verbatim contribution under the tested probes”, which appears in the abstract before the continuation-probe analysis itself has been properly motivated.

Questions for authors

Given that the 16 MBTI personas are explicitly defined by four binary dimensions, what baseline should readers use to assess whether the reported 90.8% first-order Walsh energy is genuinely surprising?

Reviewer_4UZjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks where the reliability of persona-prompted behavior in language models actually lives. It distinguishes three explanations: (i) a transferable, compositional prompt code; (ii) an item-indexed response policy that translates that code differently for different questionnaire items; and (iii) a stable latent trait whose score should be invariant to the instrument.

Critical strengths

-They actually tested whether the persona "code" travels to models it wasn't trained on. They trained a classifier on seven model families and asked it to identify the persona from an eighth family it had never seen. interesting results, showing that persona prompting genuinely does something structured and transferable, but a personality questionnaire score is the product of both the persona and the specific test you used

Critical weaknesses

The headline interaction share is partly a degrees-of-freedom artifact, and the abstract does not say so. External validity is limited to questionnaire responses under a small number of formats. The AI-use statement describes an unusually broad scope of generative-AI involvement.

Questions for authors

Can you state explicitly, in one sentence, what class of trait models remains compatible with the data, and what intervention would be needed to distinguish a fixed multidimensional latent model from the item-indexed policy account? The extensions to Big-Five, role personas, and forced-binary are each run on Qwen3.8-27B. Do the item-policy heterogeneity and the low effective rank survive on a second model in at least one of these arms? If not, how should the practical implication be scoped?

AI involvement. The AI-use statement lists study design and analysis planning among the AI-assisted contributions. Can you specify which design choices and which analyses were AI-assisted, and how the authors independently verified them against the archived data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.