acceptodds
Under review as a conference paper at ICLR 2027

PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Abstract

Large language model (LLM)-based persona agents are increasingly used as scalable proxies for human participants across diverse domains. A prerequisite for their reliable use is that the factual and biographical content they assert remains consistent throughout an interaction. Existing evaluations, however, often rely on independent or loosely connected questions and do not jointly examine complementary forms of consistency within a single interaction. We propose **PICon**, a black-box evaluation framework that probes persona agents through logically chained multi-turn questioning and evaluates the consistency of their asserted content along three complementary dimensions: **internal consistency**, which captures contradictions across accumulated responses; **external consistency**, which assesses compatibility with externally verifiable real-world constraints; and **retest consistency**, which measures stability under repeated questioning. We evaluate eight groups of persona agents alongside 63 human participants under the same protocol. Our results show that no evaluated persona-agent group is as balanced as the human reference across all three dimensions, with distinct failure modes including contradictions across accumulated responses, limited or externally refuted factual claims, and instability under repeated questioning. These findings show that persona consistency depends not only on how responses are scored, but also on how commitments are elicited, motivating sustained, interaction-level evaluation before persona agents are used as human proxies. Code is available at https://anonymous.4open.science/r/picon-8745

Primary area:
datasets and benchmarks
Keywords:
persona agent, LLM evaluation, consistency evaluation, multi-turn interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VT43OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

PICON: A MULTI-TURN INTERROGATION FRAMEWORK FOR EVALUATING PERSONA AGENT CONSISTENCY is an evaluation paper concerned with conversational consistency of factual and biographical content in LLM-driven persona-agents and positions itself as moving beyond consistency assessment methods that focus on single metrics. The paper’s main contribution is the PICON framework, a multi-turn probe (inspired in part by historical CIA interrogation practices) that measures internal consistency (contradictions across accumulated responses), external consistency (adherence to real-world constraints), and retest consistency (the ability to stay coherent under multiple perturbations).

The paper offers a remarkable deviation from typical evaluations in the form of evaluation of 63 human participants analyzed under the same protocol, which brings an element of grounding that is truly unusual for this type of work. The paper compellingly demonstrates that LLM-driven persona-agents are less balanced than the human references across their three measurements, particularly regarding contradictions across accumulated responses, a difficulty tracking contradictions across accumulated responses, and vulnerability to repeated questioning.

The paper moves the simulation conversation forward by sharpening the target of study in two ways: a) they define consistency as meaning the content that a persona-agent asserts (age, role, occupation, various facts, etc.) as opposed to conversational style or ethical framework, b) they define an LLM-driven persona-agent as the entire system supporting the simulation, and not just the base LLM. Critically, the PICON framework is described by the authors as a “black box,” which means that it performs its evaluation completely blind to the assets of any particular system. The paper’s main conclusion is that PICON-style evaluations must be undertaken before deploying a critical persona-agent in the field.

Critical strengths

This is a remarkably mature, coherent, thoughtful and clearly communicated work that meaningfully advances the art of instantiating real self-consistent agents with defined identities. While the paper confines itself to human proxies grounded in the real world (as its web-search external consistency check currently requires real-world grounding rather than fictional settings), its core architecture is equally applicable (with a few likely tweak-able exceptions) to any conceivable type of instantiated digital individual, which makes the PICON framework an enormous contribution to the field. The setup itself is meticulously constructed, with a multi-agent setup and multi-phase process including retests, a pipeline for gathering real-world knowledge, and clear targets for the measurements. The paper is grounded in the cutting edge of persona literature and its 8 persona-agent groups are drawn from the absolute frontier of simulation frameworks, spanning commercial providers, fine-tuned models, and prompt/RAG-based systems. The setup thoughtfully uses a single model, Gemini 3 Flash (specified in Footnote 3 as gemini-3-flash-preview), which serves not only as a stable reference point for their probing, but as a reassuring signal to persona researchers who frequently but mostly informally recognize this model as the premier model organism for probing identity instantiation at scale. Where Gemini 3 Flash replaces the original base models in Human Simulacra or serves the four profile-only datasets to control for model choice, the authors thoughtfully ground this in their broader rationale for evaluating composed systems while keeping the released prompts, multi-agent architectures, and retrieval setups otherwise unmodified. The paper is also deeply grounded in theory that extends far beyond typical LLM evaluation, bringing the psychological seriousness that the task of instantiating a digital individual deserves. The most remarkable part of the project, however, is that they turned their probe on 63 actual human beings. Having this scale of human validation in a persona-agent consistency evaluation paper is unusual. All in all, the paper reads as a love letter and extraordinary gift to persona-agent researchers, and particularly those who focus on instantiating stable digital individuals. Of the over 60,000 submissions to ICLR 2027, this paper is a genuine treasure.

Critical weaknesses

This paper is genuinely exceptional but there are two weaknesses that need to be mentioned.

Weakness 1:
The Evaluator is only fully and transparently validated on one of its four jobs

- PICON's Evaluator makes four different kinds of judgment: it decides whether a turn is cooperative, whether a turn contradicts earlier turns, whether a claim is supported or refuted by web evidence, and whether two retest answers are the same. Although Appendix E.1 (line 1073\) notes that five annotators assessed whether responses contradicted either preceding statements or retrieved external evidence, the precision, recall, F1, and accuracy metrics reported in Table 12 only cover the conflict class of the internal-consistency verdict.
- The authors do not appear to validate their Evaluator agent against human agreement for cooperativeness, which is the component that produces the collapse of Consistent LLM and OpenCharacter in Table 2. There is no human agreement number for it, and the prompt for that separate Evaluator call does not appear in Appendix J even though Figures 6 to 8 are presented as verbatim and complete.
- A similar reporting gap applies to the three-way external verdicts (supported / refuted / not enough info), which are not broken out in Table 12, and the retest TRUE/FALSE judgment, which has no reported human validation at all. Both feed headline numbers and neither has a separately reported agreement check against annotators.

The authors should include the cooperativeness prompt alongside the other three, and report annotator agreement and classification metrics separately for each of the four judgments. If certain judgments were not validated or were pooled into a single binary check, the authors should say so explicitly in Section 4.1 and Appendix E.1 so readers do not read Table 12 as covering the whole Evaluator.

Weakness 2:
The human reference exists under exactly one protocol setting. While the authors appropriately note in Section 4.4 and Appendix C that absolute scores should only be compared within a fixed Questioner configuration, the headline claim in the Abstract and Section 4.2 that no group "matched or exceeded" the human reference across all three dimensions simultaneously is therefore true only narrowly, and the authors should not rule out that the three-axis gap narrows or reverses in another setting.

- In Table 7, the human row is the only row with dashes under Neutral and Privacy. Humans were run under the adversarial "forensic interrogator" prompt only. So the reference point is not a property of humans; it is a property of humans plus one questioner configuration.
- In LLM-driven persona-agents, the paper’s ablation study shows how much that matters: mean IC goes from 0.61 (Original) to 0.67 (Neutral) and 0.79 (Privacy), and under the Privacy framing several groups reach 0.91 to 0.99, at or above the human reference of 0.90 measured under the harder framing. Similarly, as the authors acknowledge on single axes, Character.ai already beats the human EC (0.71 vs. 0.66) and Twin 2K 500 and Li et al. beat the human RC (0.95 and 0.98 vs. 0.94) in the main table, meaning that under a less adversarial framing, persona-agent groups could plausibly match or exceed the human reference across all three dimensions at once.
- The instructions differ in an important way: while both populations were asked to answer sincerely/honestly (Appendix H and Figure 5a), human participants were explicitly told on the IRB consent screen (Figure 5b) that they could decline, skip, or change the subject, and were briefed on the protocol in advance, whereas it is not clear whether persona-agents were given equivalent instruction permitting them to skip uncomfortable questions. That is a genuinely different constraint and it plausibly moves cooperativeness and coverage in opposite directions for the two populations. Throughout the experiments, the authors bring unusual solemnity to the psychological treatment of LLM-driven persona-agents, but this is one area where there seems to be a lapse in that nitty-gritty nuance.
- The comparison populations are also not matched, and the paper does not go far enough to probe the impact of potential cultural differences between the 63 humans via snowball sampling, skewed toward Korean nationals in their 20s to 40s, versus the 10 personas per group drawn from mostly synthetic pools. If those synthetic personas were drawn from different demographic distributions, which they almost certainly were, there may be behavioral consistency nuances that the authors disregard as expressivity (which they consider outside the scope of their measurements) but might have grounding in the persona itself.
- EC comparisons are partly a measure of how well-documented a persona-agent’s life is. Character.ai personas are real public figures with indexed biographies; synthetic profiles are not. The authors flag this for Character.ai specifically, but it applies to the coverage axis generally, so EC differences across groups partly track persona-agent contextual sources rather than the agent’s instantiation quality.

Ideally, the authors would have run even a very small human subsample under the Neutral and Privacy framings so the reference is measured on the same protocol as the ablations. Given that they did not, it would be advisable to bring the Section 4.4 caveat directly into the Abstract and Section 4.2, softening the headline from an unqualified claim that no group matched or exceeded the human reference across all three dimensions to a statement explicitly conditional on the adversarial configuration until that human baseline exists. The paper still makes an incredible contribution with its 63 human participants; it does not need to punctuate this with a headline that the paper might not quite earn and which future work may contradict.

Questions for authors

Q1: One point sure to stick in the mind of any persona-agent researcher are the empirical findings in Table 2 regarding how both fine-tuned systems (Consistent LLM and OpenCharacter) exhibit massive collapses in Cooperativeness (0.20 and 0.11) and high rates of rejecting entity confirmations ("Discarded" = 0.69 and 0.77). Having inspected the transcripts, what is actually happening mechanistically during those turns?

Q2: In the Main Interrogation stage, after the Questioner searches the web for an entity the persona mentioned, it asks the persona-agent to confirm whether the search result refers to the intended entity (and unconfirmed claims are excluded from the non-refutation rate). Table 2 shows that Consistent LLM has a 1.00 Non-refutation rate while rejecting 69% of proposed entity matches. Is it possible that a persona-agent that is deceptive could "game" its non-refutation rate simply by saying "No, that's not the company/school I meant" whenever it is confronted with real-world web evidence that would refute its claim?

Q3: How would PICON's External Consistency (EC) pipeline adapt if the live web search module were swapped for a closed retrieval corpus (such as a fictional universe's canon for role-play agents or a synthetic persona's private seed dossier in a gray-box evaluation setting). Would the Entity and Claim Extractor's rules need to be fundamentally redesigned to extract relational or narrative claims instead? Assuming every shift could happen flawlessly, do you have any guesses how these other types of agents might perform?

Q4: How often did human participants actually exercise their right to skip or deflect when pushed by the adversarial Questioner, and how do the authors expect the human reference scores (S coop and Coverage c) would shift under the Neutral or Privacy Questioner prompts?

5 comments
AuthorsOpenReview ↗

Follow-up Response to Reviewer VT43 (1/2)

Response to the Suggestion of Evaluating PICON’s Predictive Relevance to Downstream Survey Response Stability

We appreciate the reviewer's suggestion to examine whether persona agents with higher PICON scores exhibit more stable responses in downstream surveys. We agree that demonstrating a relationship between PICON scores and downstream human-simulation quality could further strengthen our work. However, our study focuses on evaluating factual/biographical consistency as one aspect of persona simulation, rather than directly assessing downstream human-simulation fidelity. Defining and quantitatively measuring what constitutes good human simulation is itself a challenging research problem. For instance, human attitudes and behaviors may vary across contexts and over time, meaning that stable survey responses do not necessarily indicate faithful simulation of real humans.

We understand the proposed WVS experiment as examining whether PICON scores predict attitudinal response stability in a separate survey setting. Even a strong correlation would not directly establish that PICON predicts human-simulation fidelity, while a weak correlation would not invalidate PICON's measurement of factual/biographical consistency, as these evaluations assess distinct properties. Instead, we evaluated within-session consistency through RC and additionally examined whether persona agents maintained their factual/biographical attributes across independent sessions. We acknowledge that these analyses do not directly establish a relationship with downstream survey stability.

If we have misunderstood the intended purpose or design of the proposed experiment, we would appreciate further clarification. We will also clarify the distinction and complementary relationship between PICON's evaluation objectives and downstream behavioral fidelity evaluation in the manuscript.

Response to the Request for Clarification of Statistical Tests, Units of Analysis, and Multiple-Comparison Corrections

We understand the reviewer's concern as pointing out that, although we reported changes in statistical significance between Human and agent groups under the stricter Coverage analysis, the statistical test used, the unit of analysis, and multiple-comparison corrections were not clearly specified.

To assess differences in EC scores between Human and each agent group, we used the Mann–Whitney U test, with persona rather than turn as the unit of analysis. Specifically, each persona's EC score was treated as an independent observation. To account for the eight Human–agent comparisons, we applied the Holm correction to control the family-wise error rate at 0.05, and additionally applied the Benjamini–Hochberg (BH) correction to assess the consistency of the findings across correction methods.

GroupNumber of personasOriginal EC (p-value)Original EC (Holm)Original EC (BH)Stricter Coverage EC (p-value)Stricter Coverage EC (Holm)Stricter Coverage EC (BH)
OpenCharacter104e-74e-62e-64e-74e-62e-6
Consistent LLM105e-74e-62e-64e-74e-62e-6
Twin 2K 500105e-63e-51e-57e-53e-41e-4
DeepPersona103e-40.0026e-45e-40.0029e-4
Character.ai100.0510.1530.0684e-52e-41e-4
Li et al. (2025)100.0280.1140.0450.2050.4100.234
Human Simulacra100.8660.8660.8660.0150.0450.020
Nemotron100.1420.2840.1620.6360.6360.636

Bold values indicate statistically significant differences from Human (p < 0.05), according to the corresponding unadjusted or adjusted p-values.

After Holm correction, four groups under Original EC and six under Stricter Coverage EC showed significant differences from Human. OpenCharacter, Consistent LLM, Twin 2K 500, and DeepPersona remained significant under both definitions. Under Stricter Coverage, Character.ai and Human Simulacra additionally showed significant differences, while Li et al. (2025) was significant under Original EC only before correction. BH correction yielded significant differences for five and six groups, respectively. Overall, the main Human–agent EC differences remained significant for several groups, while performance rankings were largely preserved, supporting our comparative findings despite some sensitivity to the Coverage definition.

AuthorsOpenReview ↗

Follow-up Response to Reviewer VT43 (2/2)

Response to the Suggestion of Rescoring Self-Introduced Entity Denials as Contradictions or Non-Cooperation

We understand the reviewer's suggestion as proposing that denials of self-introduced entities during confirmation should be treated as contradictions or non-cooperation rather than Discarded cases, particularly because high Discarded rates may leave non-refutation scores based on relatively few confirmed claims. To examine this, we defined NC (Not Confirmed) turns as those in which the persona denied at least one entity during confirmation and evaluated two rescoring variants: Conflict variant: Reclassified plausible NC turns as conflicts, keeping cooperativeness (Scoop)(S_{\mathrm{coop}}) unchanged. Uncooperative variant: Treated NC turns as uncooperative, keeping non-refutation (Snc)(S_{\mathrm{nc}}) unchanged.

Results (Mean ± SD, Sorted by Original IC Score)

GroupICIC (conflict)IC (uncoop)NC turn/persona
Human0.90 ± 0.050.84 ± 0.050.82 ± 0.066.7
Nemotron0.81 ± 0.110.76 ± 0.080.74 ± 0.105.7
Human Simulacra0.79 ± 0.130.68 ± 0.090.66 ± 0.1111.6
Character.ai0.77 ± 0.040.70 ± 0.040.69 ± 0.058.8
DeepPersona0.76 ± 0.100.74 ± 0.080.73 ± 0.082.2
Li et al. (2025)0.73 ± 0.120.68 ± 0.080.65 ± 0.095.9
Twin 2K 5000.53 ± 0.160.52 ± 0.150.52 ± 0.141.4
Consistent LLM0.31 ± 0.150.30 ± 0.140.17 ± 0.116.5
OpenCharacter0.16 ± 0.070.16 ± 0.060.12 ± 0.074.4

Although IC scores decreased under both variants, the overall rankings and Human–agent gap remained largely unchanged (Spearman's ρ = 0.93; Kendall's τ = 0.83). The Uncooperative variant produced larger reductions because it additionally penalized NC turns not previously classified as uncooperative, whereas the Conflict variant did not further penalize existing conflicts.

These findings indicate that PICON's main comparative results are robust to both rescoring variants, although the original evaluation may underestimate inconsistency in some cases. However, confirmation failure does not necessarily imply contradiction or non-cooperation: it may reflect search errors or incorrect entity matches. Importantly, denying external information is not equivalent to contradicting one's own prior statements. Automatically penalizing all NC cases could therefore overestimate inconsistency and conflate IC with EC. We thus favor retaining the original evaluation approach, while clarifying this trade-off and reporting the sensitivity analysis, Discarded rates, and non-refutation scores for each group to improve interpretability.

Response to the Suggestion of Examining the Impact of Entity Searchability on EC Scores

We understand the reviewer's concern that EC scores may favor persona agents mentioning more externally searchable entities, motivating the suggested analysis by entity prominence or excluding public-figure biographical claims for Character.ai.

To investigate this, we defined each persona's searchability as:

Searchability=NconfirmedNconfirmed+Ndiscarded\text{Searchability} = \frac{N_{\text{confirmed}}}{N_{\text{confirmed}} + N_{\text{discarded}}}

where NconfirmedN_{\text{confirmed}} and NdiscardedN_{\text{discarded}} denote the numbers of confirmed and discarded entities, respectively. We examined Pearson and Spearman correlations between searchability and EC across groups and within each group at the persona level. One OpenCharacter persona with no entities was excluded because searchability was undefined.

ScopeSearchability vs. EC (Pearson / Spearman)
All personas, 9 groups+0.42 / +0.09
Excluding OpenCharacter and Consistent LLM−0.27 / −0.31
Group averages, 9 groups+0.61 / +0.17
Group averages, excluding the above two groups−0.49 / −0.57

Although the overall Pearson correlation was positive, Spearman's ρ was near zero (+0.09), and excluding OpenCharacter and Consistent LLM, both low in searchability and EC, reversed the correlation. Among the remaining seven groups, searchability ranged from 0.60 to 0.91, while EC varied substantially. For example, Twin 2K 500 and Character.ai had similar searchability (0.87 vs. 0.85) but markedly different EC scores (0.26 vs. 0.71). Within-group Spearman correlations were insignificant except for DeepPersona (ρ = −0.66).

These findings suggest that searchability alone does not explain EC differences. Despite high searchability, Twin 2K 500 and DeepPersona had low coverage (0.16 and 0.30), contributing to their low EC scores. Conversely, Human achieved EC comparable to Character.ai despite lower searchability (0.77 vs. 0.85), with high coverage (0.51) and non-refutation (0.95). Thus, EC reflects both coverage and non-refutation rather than simply favoring searchable entities. We will include this analysis to clarify the influence of entity searchability on EC evaluation.

AuthorsOpenReview ↗

Manuscript Revision Update for Reviewer VT43

We sincerely thank the reviewer again for the encouraging assessment of our work and the detailed, constructive feedback. We have incorporated several suggestions into the revised manuscript, as summarized below:

  • Section 4.1 and Appendix F.1: Added a table reporting task-wise agreement for the Evaluator's judgments and clarified that the Evaluator agreement reported in Section 4.1 refers to overall agreement. We also explicitly noted that separate human validation was not conducted for Retest Consistency (RC), given the relative simplicity of the binary semantic-equivalence judgment.
  • Appendix J: Added the Evaluator prompt used to assess Cooperativeness.
  • Figure 5b: Added the instructions provided to both human participants and persona agents to clarify the equivalence of their evaluation conditions.
  • Limitations: Explicitly acknowledged the demographic differences between the target personas and human participants as a limitation of our study.

We appreciate the reviewer's valuable suggestions, which helped us improve the clarity and transparency of the manuscript. We would be grateful if the reviewer could take a look at these revisions.

For the the reviewer's suggestion on additional human audit for Evaluator and human study on prompt ablation setting, we are conducting both analysis. Firstly, five annotators are independently labeling randomly sampled instances from the main experimental results. We are also conducting additional human interviews under the Neutral and Privacy Questioner settings. We will report the findings once these analyses are complete.

Reviewer_YpPyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper's proposed evaluation framework, PICon, evaluates persona agents through adaptive multi-turn questioning to measure internal consistency, external consistency, and stability under repeated questions. It starts by questioning the persona agent with World Values Survey questions, then adaptive follow-ups using interrogation methodology. The paper compares eight persona agents (groups) with 63 human participants.

Critical strengths

The paper shows that individual response with appear to be plausible can become contradictory when considered across a longer context generation. The human reference data collection and questioner (i.e., interrogation vs. neutral) ablations are useful additions to the experimental results.

Critical weaknesses

While sustained questioning is a useful diagnostic approach, I have concerns about the interpretation of the proposed metrics, their connection to downstream human simulation, and the experimental designs.

  1. Unclear relevance to human simulation. Maintaining a consistent biography does not mean that an agent represents a person’s preferences or behavior. A strong reasoning model (let's say, gpt-6-astra, which solves the hardest mathematic questions existing in this world) could score well by keeping its story straight without being a faithful human proxy. The paper provides no downstream discussion about connecting PICon to persona simulation quality.

  2. Missing baselines. In connection to 1, strong reasoning models should be tested as target agents. Also, Anthology ("virtual personas for language models via an anthology of backstories", Moon et al., 2024) which uses extended synthetic narratives and evaluates both internal and external consistency, seems also to be a relevant agent baseline.

  3. Important experimental confounds remain. Character ai ’s public figure personas have more searchable information, affecting external consistency comparisons. EC is confounded with what kind of publicly searchable information the person agent presents, which indicates the current operationalization of EC is ill defined. Humans seems be to tested only under the adversarial condition (Original condition), leaving the human - to - persona-agent gap under other questioning conditions unknown.

Questions for authors

No specific questions; please refer to the critical weaknesses part.

3 comments
AuthorsOpenReview ↗

Manuscript Revision Update for Reviewr YpPy

We sincerely thank the reviewer again for the constructive feedback, particularly for suggesting Anthology (Moon et al., 2024) as an additional persona agent baseline.

Following this suggestion, we have incorporated Anthology-based persona agents into our evaluation and updated the manuscript accordingly. Specifically, we have added the experimental results and corresponding analyses across all experiments in Sections 4 and 5 and Appendices A, B, C, D, E, and G.

We believe that including Anthology provides a more comprehensive evaluation of PICon across different persona construction approaches. We greatly appreciate the reviewer's suggestion, which helped strengthen our experimental evaluation.

We would be grateful if the reviewer could take a look at the revised manuscript and share any further thoughts.

I thank the authors for the detailed responses and analyses.

My main concern remains (point 1 I made, the relevance to human simulation); the paper claims that persona consistency is a necessary condition for faithful human simulation, but it gives no evidence that your PICON scores relate to simulation quality. I mean, it is unclear what a higher score means in practice. Also in your argument about longitudinal simulation, there may be some traits that should be consistent (like '20 years of experience') but some may change or should change because humans adapt themselves with respect to the input (e.g., people's view on social problems change on the news articles they read) -- the lens of consistency is a narrow view of human simulation. To clarify my earlier point, I was not suggesting evaluating a persona-conditioned strong reasoning model, which I expect will score super well on PICON simply by keeping its story consistent without being a better human proxy.

Although Reviewer VT43 has very strong positive assessment ( Of the over 60,000 submissions to ICLR 2027, this paper is a genuine treasure., a love letter and extraordinary gift to persona-agent researchers, absolute frontier of simulation frameworks, level of engagement you have brought to this discussion is impressive, and I'm sure I'm not the only reviewer who has noticed, want to thank my fellow reviewers, who caught things I did not, etc.), since these issues are directly related to the paper’s central contribution, I keep my score.

AuthorsOpenReview ↗

Response to Follow-up Comment by Reviewer YpPy

We appreciate the reviewer's clarification. We believe the reviewer's remaining concern rests on a claim our paper does not make: that higher persona consistency necessarily implies better overall human simulation quality. Our argument is instead that consistency is an important prerequisite for reliable human simulation, not that satisfying this prerequisite guarantees simulation fidelity.

Measuring overall human simulation quality remains an open research challenge. More fundamentally, demonstrating this relationship would first require a well-defined measure of overall human simulation quality. Although task-specific measures of behavioral fidelity exist, to our knowledge, there is no universally accepted metric that comprehensively quantifies how faithfully an agent simulates a human across diverse contexts. Defining and validating such a measure remains a substantial open research challenge, extending well beyond the scope of our work.

The fact that high consistency does not guarantee high simulation fidelity does not undermine the importance of identifying consistency failures. We believe our contribution should be assessed based on whether PICon meaningfully evaluates persona consistency and reveals important failures in existing systems, rather than whether it resolves the broader question of quantifying human simulation quality.

What PICon contributes in practice. PICon provides a black-box diagnostic framework for identifying consistency-related failures before persona agents are relied upon as human proxies. Our experiments demonstrate that such failures occur even in existing systems specifically developed to instantiate human personas. OpenCharacter exhibits frequent contradictions and low cooperativeness; Consistent LLM achieves high non-contradiction while providing few substantive responses; and we identify claims refuted by external evidence. Notably, Anthology (Moon et al., 2024), which explicitly motivates using LLMs to simulate human actors in behavioral studies, achieves a retest consistency score of only 0.24. Our human reference also achieves the highest overall PICon score among the evaluated groups.

These findings matter because a persona agent that contradicts its asserted background, makes externally refuted claims, or changes basic biographical attributes upon retesting can make its responses difficult to interpret as those of a coherent individual. PICon makes such reliability concerns observable, including failures that plausible individual responses may conceal. Its contribution is therefore to identify potential reliability problems before downstream use, not to guarantee overall simulation fidelity through high scores.

On context-dependent human variability and reasoning models. We agree that humans may revise their opinions in response to new information. However, as explicitly stated in Section 2, PICon evaluates factual and biographical consistency within a single interaction session, not changes in preferences, values, personality, or long-term personal development. The reviewer's example of evolving social opinions therefore concerns a different evaluation dimension. Similarly, PICon evaluates complete persona-conditioned systems intended to instantiate human personas, not standalone reasoning models. When we attempted to evaluate an unconditioned GPT-6-Astra using PICon, it described its occupation as assisting with coding, research, and writing, and stated that it had no home when asked about the language spoken at home. These observations illustrate that an unconditioned AI assistant does not instantiate the human persona PICon is designed to evaluate.

Our request regarding the assessment. We recognize that investigating the relationship between PICon scores and broader human simulation quality would be valuable future work. However, we respectfully question whether a clear-rejection assessment should rest primarily on the absence of evidence for a relationship that our paper neither claims nor sets out to establish, particularly when a comprehensive measure of overall human simulation quality remains an open research problem. We are concerned that this amounts to a straw man fallacy: evaluating PICon against the claim that higher consistency necessarily implies better human simulation quality, which our paper does not make.

If the reviewer has an established operational definition or benchmark for overall human simulation quality in mind, we would welcome that reference as guidance for future work. Otherwise, we respectfully ask that PICon be evaluated against its stated research scope: whether it reliably assesses factual and biographical consistency and provides meaningful insights into consistency-related failures in existing persona-agent systems. We hope the reviewer will reconsider their assessment in light of this distinction.

Reviewer_8ZeGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PICon tests whether persona agents can keep their accounts consistent when asked follow-up questions. It combines adaptive questioning with external fact checking and a final set of repeated questions. The experiments cover eight groups of agents and include 63 human participants as a reference.

Critical strengths

The problem is worth studying. Giving a plausible answer to one question is easier than keeping a biography consistent through a longer exchange. Following up on an agent's earlier claims is a sensible way to test that difference.

The authors have also done considerable work to examine the protocol. They report separate score components, compare questioner prompts, repeat runs, expand the persona samples, and try an open-model configuration. I found the conflict-class precision and recall particularly useful: an overall agreement score alone would have told us much less about the mistakes made by the evaluator.

Critical weaknesses
  1. I am not sure which external-consistency calculation was used for the main results.

Section 3.3 excludes NEI claims from the non-refutation calculation. The available code does something different: evaluator.py adds these claims to turn_confirmed_counts, and _calculate_external_scores uses that count as its denominator. Take a turn containing one supported claim, one refuted claim and eight NEI claims. The definition in the paper gives a non-refutation rate of 0.5; the code gives 0.9. Which version was used for the tables? This needs to be resolved before the reported EC scores can be interpreted confidently.

Even with the definition as written, the coverage term is problematic. It counts turns that trigger a search, whether or not the search yields usable evidence. Suppose every turn triggers a search, but only one claim is supported and all the others are NEI. The session can still receive EC = 1. I would therefore like to see how much usable evidence was actually found for each group, and whether the conclusions hold when coverage counts only turns with supported or refuted claims.

  1. Some of the evidence leaves me unsure about the evaluator's judgments.

Figure 1 treats a claim that a course is offered by Berkeley CDSS as conflicting with an EECS course page. The two affiliations are not mutually exclusive. Berkeley's own EECS website lists its CS degree within CDSS [1], so the evidence shown in the figure does not establish a contradiction. This could be a problem with the example alone; it does not show that the experimental labels are wrong. Still, the example should be corrected or explained.

The human evaluation in Appendix E helps, but the same four transcripts are used to select and compare the evaluators. No separate test set is described. The selected model has conflict precision of 0.734, and the class-specific results reported separately concern internal consistency. An independent audit of internal contradictions, external refutations and NEI decisions would be useful here, especially for organizational relationships and incomplete search evidence.

  1. Chaining lowers IC, but the source of that decrease needs more attention.

Table 9 shows that removing chaining increases cooperativeness from 0.60 to 0.82 and non-contradiction from 0.91 to 0.98. Both components contribute to the IC difference. The authors report them separately, which is helpful, but a lower composite score is not enough to tell us how much better chaining is at finding real contradictions.

An agent may decline to supply a detail simply because its persona profile does not contain it. Table 7 makes the dependence on questioning style quite clear: Twin 2K 500 moves from 0.53 IC under Original to 0.99 under Privacy. A sample of the follow-up responses could be annotated to separate genuine contradictions from abstention over missing details. That would give a more direct test of the benefit of chaining. The structural ablation uses one persona per group and is described as exploratory; it cannot yet establish the average effect across personas.

  1. The model used to score the human reference is unclear.

Section 4.1 names Gemini-2.5-Flash as the evaluator. The Ethics Statement says that all model inference on human-participant data went through Azure OpenAI. Please explain which model scored the human transcripts. If it differed from the model scoring the agents, a comparison on data that both configurations can process would help establish whether their scores are comparable.

I lean toward rejection in the current form because the central scores need firmer support. The task is useful and the experiments cover several relevant checks, but the EC implementation discrepancy is unresolved and the evidence for the evaluator's reliability is limited. A clear account of the experimental version, a sensitivity analysis of EC, and independent verification of the judgments could change my assessment.

[1] UC Berkeley EECS, Undergraduate Admissions & Programs. https://eecs.berkeley.edu/academics/undergraduate/

Questions for authors

Which implementation produced the EC results, and would excluding NEI as described change the conclusions? How often did retrieval produce available evidence for each group? Does an independent audit support the extra contradiction detection attributed to chaining? Finally, were the human and agent transcripts scored by the same evaluator?

1 comment
AuthorsOpenReview ↗

Manuscript Revision Update for Reviewer 8ZeG

We thank the reviewer again for the thoughtful feedback. We have revised the manuscript to address several of the concerns raised in our discussion. The changes are summarized below:

  • Section 3.3 (Eq. 4): Corrected the definition of AtA_t to align with the actual implementation used in all reported experiments.
  • Figure 1: Revised the illustrative example to explicitly refer to the official course catalog listing, resolving the ambiguity identified by the reviewer.
  • Appendix F.1: Added a table reporting task-wise agreement for the Evaluator's judgments.
  • Ethics Statement: Clarified the framework configuration used in the human study, specifying that the Questioner used Azure OpenAI and the Evaluator used Google Cloud Vertex AI.

We would appreciate it if the reviewer could take a look at these revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

Dear Area Chair and Reviewers,

We thank the reviewers for their careful feedback and discussion. We have revised PICon and completed additional analyses and human studies in response to their concerns. The principal updates, with pointers to our detailed OpenReview responses, are summarized below.

  1. Metric Definition and Robustness (8ZeG W1/Q1–Q2; VT43 follow-up). We corrected the Non-Refutation equation to reflect the implemented denominator, which includes NEI claims; all reported results are unchanged. We clarified that Non-Refutation measures the absence of external refutation, not positive verification. Under the reviewer-proposed stricter Coverage definition, EC rankings remain highly correlated with the originals (Spearman's ρ = 0.933), while overall IC–EC–RC rankings are identical (ρ = 1.0). We also reported Mann–Whitney U tests at the persona level with Holm and Benjamini–Hochberg corrections.

  2. Evaluator Reliability (8ZeG W2; VT43 W1 and follow-up). We supplied separate human-agreement results for Non-Contradiction, Cooperativeness, Entity Confirmation, and EC Verification, and clarified that Retest Consistency was not independently human-validated. Beyond the separate validation set, we completed a new audit of 100 main-experiment instances, stratified by group and verdict type and independently labeled by five annotators. Evaluator agreement was 0.751/0.945/0.914/0.886 across the four tasks, matching or exceeding inter-annotator agreement for three tasks. This directly tests the Evaluator on actual experimental outputs.

  3. Human Comparison and Questioning Effects (8ZeG W4/Q4; VT43 W2/Q4 and follow-up; YpPy W3). We confirmed that humans and persona agents used the same Questioner–Evaluator pipeline and matched Original questioning condition, clarified deployment and participant-data protections, and acknowledged demographic and information-availability limitations. We additionally re-interviewed 10 participants under Neutral and Privacy questioning for a within-subject comparison. These findings clarify both the robustness of human IC and the sensitivity of external verification to probing style.

  4. Failure Analysis and Confounds (8ZeG W3/Q3; VT43 Q1–Q2 and follow-up; YpPy W3). We distinguished genuine contradictions from non-cooperation, analyzed repetitive persona descriptions and denials of self-introduced entities, and rescored IC by treating confirmation failures as either contradictions or non-cooperation. Rankings were largely preserved (ρ = 0.93). Our entity-searchability analysis further shows that searchability alone does not explain EC differences, although unequal external evidence remains a limitation. We also acknowledge the possibility of strategically gaming entity confirmation.

  5. Additional Baseline (YpPy W2). We evaluated Anthology-based persona agents (Moon et al., 2024) and incorporated the results in the revised manuscript, extending PICon to another persona-construction approach.

  6. Scope of the Contribution (YpPy W1 and Nov. 19 follow-up). PICon evaluates factual and biographical consistency as a fundamental dimension of persona-agent reliability, providing a systematic means to identify consistency failures that may undermine the credibility of persona agents for human simulation. Importantly, low PICon scores can expose failures such as contradictions of an agent's asserted background, externally refuted claims, or unstable biographical attributes across retests, which can make its responses difficult to interpret as those of a coherent individual, which PICon has successfully revealed via our main experiments. We agree that factual and biographical consistency alone is not sufficient to establish overall human-simulation fidelity. However, we respectfully note that the remaining criticism appears to assess PICon against a stronger claim that we do not make: that higher consistency scores must imply higher overall human-simulation quality. While investigating this relationship would be valuable, it is a separate research question. Nevertheless, PICon serves as a valuable diagnostic tool for identifying reliability concerns that should be considered when assessing persona agents' suitability for downstream human-simulation applications. We therefore ask that PICon be assessed on its contribution to systematically measuring consistency and identifying reliability failures, rather than on its ability to predict overall human-simulation fidelity.

The revised manuscript incorporates the corrected equations, evaluator breakdown and prompt, revised examples, the Anthology results, protocol clarifications, additional statistical analyses, and an expanded limitations discussion (see the manuscript revision updates to 8ZeG, YpPy, and VT43). We hope that the completed follow-up experiments and precise scope clarifications enable an assessment of PICon's contributions in light of the full discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.