- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Personas in Pieces: Factorial Prompt Geometry and Item-Indexed Response Heterogeneity in Language Models claims that the growing body of research into how personality prompting affects model behavior requires a more detailed explanation of the impact of prompt wording and specific item perturbation. The authors claim to make five contributions:
- A held-family-out test of transferable persona information, i.e., a nearest-centroid classifier computed from the domain profiles of 7 different language model families, then applied to an 8th held-out model family.
- Quantified compositionality "with an exact Walsh expansion," by which they mean a mathematical breakdown of how the four binary components of the MBTI personality schema factor into the domain profiles produced by language models assigned specific MBTI personalities.
- A measure of "contextual expression with an orthogonal response-tensor decomposition," which translates into simpler language as a standard three-way ANOVA across models, personas, and questionnaire items. They measure how much variance is accounted for by the persona-by-item interaction.
- A demonstration of "how aggregation turns policy heterogeneity into key-sensitive scores and rank changes," which means a comparison between standard scale averaging (where all items are weighted equally) and "equal-key" averaging (where positively worded and negatively worded item subsets are each weighted 50/50, regardless of how many items are in each subset). The authors show that on scales with unequal numbers of forward and reverse items, reweighting shifts scores slightly and flips 10.2% of untied model rankings in the unprompted Default condition. (Note: how this is framed in the paper will be discussed later in this review.)
- A replication of "both levels on two family-external models with template, wording, memorization, taxonomy, and response-format controls," by which they mean that they re-ran the experiment on another two models with a variety of attempts to mask or modify wording and response formats to test the robustness of their findings.
The authors conclude that their prompts have a specific transferable structure at the aggregate level, while item-level expression is a joint function of the prompted persona and the specific questionnaire instrument.
Critical strengths
The empirical data collection is genuinely large in scale (375,700 primary responses across 20 models from 8 families and 4 inventories, plus two external replication models) and includes several perturbation arms (paraphrasing, template changes, forced-binary formatting). Releasing this data, wherever provider conditions allow, would be a contribution of a real asset.
Critical weaknesses
Weakness #1: The central statistical results do not support the conclusions, and the paper's own analysis contradicts its Abstract.
a) The headline "item-indexed response policy" is largely an artifact of pooling 17 distinct personality domains into a single ANOVA.
The authors run one three-way ANOVA across all 221 items (146 Likert items and 74 binary items separately) across all four inventories simultaneously (Equation 2). But those items measure 17 different domains: Extraversion, Openness, Agreeableness, Conscientiousness, Neuroticism, Narcissism, Psychopathy, the Lie scale, etc. When you tell a model it is an ESTJ, it should go up on Extraversion items and down on Openness items. Nothing about that requires an item-specific "response policy." In a pooled ANOVA across a multi-trait battery, a trait-specific shift across domains registers automatically as a Persona x Item interaction. The authors report that interaction at 48.7% of Likert variance (52.0% binary) and present it as evidence of item-specific heterogeneity. They then admit in Section 3.4 that this interaction has an effective rank of 3.9, which aligns with the four MBTI letters, and that per degree of freedom its squared norm is only 1.1x the model main effect (1.7x for binary). When they switch to five-dimensional Big Five prompts in their extension (Table 7), that effective rank becomes 5.12, which again aligns with the five Big Five factors. In other words, the dimensionality of the interaction tracks the number of trait dimensions they prompted with, which is entirely what you would expect when pooling across traits. While the authors briefly mention a correlation with "within-domain interaction share" in Section 3.4, the primary 48.7% and 52.0% headline figures are computed on the pooled multi-domain grid rather than decomposed within individual single-factor scales.
b) The "stable latent trait" null tests an unrealistic one-dimensional reduction of a multidimensional construct, and the authors admit in the body text that a multidimensional latent model fits their data while claiming the opposite in the Abstract.
In Section 3.8, the authors test whether a single one-dimensional trait coordinate can account for the Persona x Item interaction across all 16 MBTI personas and all 17 domains. However, the MBTI is a four-dimensional system, and the Big Five is a five-dimensional one. Even this one-dimensional null absorbs 68% of the Likert interaction and 81% of the binary interaction, as they report. When they test a four-dimensional latent model that matches the four-factor structure of the MBTI prompt, they report that absorption rises to 86% to 101% (and up to 102% with four principal components) of the observed interaction. They explicitly state in Section 3.8 that "a fixed multidimensional latent model with heterogeneous loadings remains compatible with these observations." Yet the Abstract states that "the experiment does not identify a stable-trait interpretation of the resulting scores."
In psychometrics, personality inventories are multidimensional by construction, and items are expected to differ in their factor loadings and intercepts. That is the premise of factor analysis and multidimensional IRT. Asking whether a single dimension can explain responses across 17 domains tests a strawman version of a trait model, making it unsurprising that it does not fully account for the data. Because the authors concede in Section 3.8 that a multidimensional model with heterogeneous loadings is compatible with their observations, the headline claim in the Abstract would lead a reader to a very different conclusion than what the body of the paper supports.
c) The "10.2% rank reversal" is a misleading headline.
The Abstract reports that equal-key scoring "reverses 10.2% of untied model rankings" in a context that implies this instability affects persona evaluation. In Section 3.5, however, the text clarifies that this figure refers to rankings of the 20 models against each other in the unprompted Default condition. Beyond that, Table 5 and Figure 5c show that this average is heavily influenced by an outlier scale (SD3 Narcissism, which flips over 30% of model pairs despite a mean score shift of only 0.027 on a 0-to-1 scale, roughly a tenth of a Likert point) applied to default model scores that are already tightly bunched together. Indeed, the authors report that model identity accounts for only 0.36% of Likert variance, which explains why tiny shifts reorder unprompted models: the between-model differences are minuscule to begin with. Meanwhile, their own persona decoding remains at 99.1% under equal-key scoring, so the persona profiles themselves are barely affected. This is a statistic that is technically true of one specific subset of the dataset, but it does not support the broader narrative implied by the Abstract.
Weakness #2: The paper has weak grounding in prior literature and falls short of basic scholarly, reproducibility, and presentation standards.
a) The paper's framing overlooks directly relevant recent literature, such as the ICLR 2026 paper "From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers" by Liu et al.
Liu et al. already examined the dissociation between factor-level alignment and item-level heterogeneity, and benchmarked it against human ground truth across a wider array of psychological questionnaires beyond personality. One of their primary findings occurred during item-level analysis. From that paper's abstract: "For information selection, LLMs identify the same key personality factors as trained algorithms, though they fail to differentiate item importance within factors." Because Liu et al. included human ground truth as a comparison, they were able to contextualize item-level variability against human data. Failing to engage with prior work at the same venue is symptomatic of a thin literature review that does not situate the paper's claims properly.
b) The paper does not engage with the foundational personality literature mapping these constructs to one another.
The classifiers used to test "transferability" are recovering a well-established psychometric mapping. Because the 16 MBTI profiles differ primarily along the four Big Five domains with which the MBTI is known to overlap, a simple rule based on the published psychological literature (e.g., mapping E/I to Extraversion, S/N to Openness, T/F to Agreeableness, and J/P to Conscientiousness) would recover the labels without training a classifier across model families. That a seven-family-trained nearest-centroid classifier reaches 99.7% on the eighth reflects the fact that all tested model families have learned the standard mapping that has been documented in personality psychology for nearly four decades (e.g., McCrae & Costa, 1989).
Similarly, the constructs in ZKPQ, EPQR-A, and SD3 have decades of published cross-validation against the Big Five (e.g., Zuckerman et al., 1993). It is well documented in the literature that E tracks Extraversion and N tracks Openness, which is also the basis of popular online assessments such as 16personalities.com. The authors do not cite this foundational literature. Furthermore, treating multi-scale personality data as if a trait account requires a single common dimension (Weakness #1b) or uniform cross-battery item shifts (Weakness #1a and the sign-consistency metric in Appendix F) reflects a disconnect from how multidimensional personality inventories are designed.
c) The conceptual framing does not engage with how representations are formed during LLM pre-training and inference.
Shared pre-training data readily explains the consistency across constructs and model families: because decades of psychometric discussions and archetype descriptions are present in pre-training corpora, mapping between Big Five, MBTI, ZKPQ, and EPQR-A is practically as straightforward for capable language models as translating between Norwegian and Swedish.
The paper's framing of an "item-indexed response policy" versus a "prompt code" also obscures how transformer inference operates. Because transformer-based language models are stateless across separate calls, every output is naturally computed as a conditional function of both the persona prompt tokens and the specific item tokens in the context window on each forward pass. Framing this conditional computation as a surprising alternative to a human-like latent trait sets up an artificial dichotomy.
d) The bibliography contains severe formatting errors and omissions.
Most notably, the bibliography cites Jacob Andreas' 2019 ICLR paper Measuring Compositionality in Representation Learning as "Jacob Andreas et al. ... volume 1, pp. 3, 2019," adding a non-existent "et al." to a single-author paper and including inaccurate volume and page numbers.
Other entries are incomplete, such as: "Delroy L Paulhus. Measurement and control of response bias. 1991." This entry omits the book title, editors, publisher, and page numbers (Paulhus, D. L., 1991, Measurement and control of response bias, in J. P. Robinson, P. R. Shaver, & L. S. Wrightsman, Eds., Measures of personality and social psychological attitudes, pp. 17-59, Academic Press).
Additionally, while the primary experimental design revolves around the 16 MBTI personas (and Big Five factorial personas in an extension), foundational sources for the Myers-Briggs Type Indicator and its relation to the Five-Factor Model are not cited. The state of the bibliography falls well below expectations for a conference submission and supports my recommendation to reject.
e) The presentation relies on unnecessarily dense mathematical notation ("mathiness") and undefined variables.
The manuscript contains numerous equations with unlabeled variables. In Section 2 (Equations 1, 2, and 5), the authors do not define the subscripts (m, p, i) or component terms (a_m, b_p, c_i) in the prose, and in Section 4 (Discussion) they introduce a new symbol (Δc_i) and numerical values that are only defined later in Appendix F. Standard three-way ANOVA and a 2^4 factorial contrast are presented as an "orthogonal response-tensor decomposition," "Walsh expansion," and "Parseval's identity," which obscures routine variance decompositions behind signal-processing terminology. The paper also introduces bespoke terms ("prompt code," "item-indexed response policy") in the Abstract and Introduction before defining what they measure empirically, resulting in an Abstract that is dense with numbers but difficult to interpret.
f) Although the paper focuses on prompt geometry and item wording, neither the prompt templates nor the modified items are shown in the manuscript or Appendix.
Unless an external repository is provided, the experiments cannot be reproduced from the paper and Appendix alone. Because the study uses standard public inventories and synthetic paraphrases rather than sensitive human data, the exact texts should be included in the Appendix. Not a single verbatim prompt template appears in the paper or Appendix, including the Default prompt, the 16 MBTI prompts, the "masked" template that removes MBTI letters, the reworded template, the 32 Big Five prompts, the 16 natural-language role personas, or examples of the paraphrased items.
g) The manuscript shows signs of insufficient human proofreading and quality control.
AI can be a fantastic resource during intense, complicated research, and make valid contributions even at the level of design, framing, and hypothesis generation. Human control over the collaborative process, however, is paramount for any research in which AI cannot be credited as a co-author, which as of 2026, means that humans must be holding the reign for work submitted to almost every reputable conference and journal.
Several aspects of the manuscript suggest that manual review and quality control were insufficient:
- The bibliography errors noted in (d) above.
- Corrupted and incomplete sentences left in the text
- Critical caveats that qualify or undercut headline claims are placed at the end of Sections 3.4 and 3.8 and in Appendix G, without updating the corresponding claims in the Abstract and Introduction.
- References to apparent internal local file paths (such as 'code/qwen_tier1_runner.py' and 'code/results/iclr/round4/') in the Appendix in place of self-contained descriptions or tables of the prompts. If these refer to a supplemental package, that package was not available for review during this assessment.
h) The five stated contributions read more like a list of analysis steps than substantive scientific findings.
The brief Discussion and Conclusion (roughly two-thirds of a page combined) do not adequately synthesize or defend them.
Questions for authors
- Can you provide the exact verbatim text of the Default prompt, the 16 MBTI prompts, the "masked" template (that removes MBTI letters and type names), and the 16 natural-language role personas?
- Are you willing to release the output, since that would be a genuine contribution and there should be no ethics concerns given the fully synthetic data (pending, of course, provider output terms)?
- You report that equal-key scoring reverses 10.2% of untied model rankings in the unprompted Default condition. What percentage of untied persona rankings (across the 16 MBTI prompts within a model) are reversed under equal-key scoring on the target domains?