acceptodds
Under review as a conference paper at ICLR 2027

Let Minstrel Tell Your Story: Reconstructing Character Profiles from Descriptions for Role Playing

Abstract

Role-playing agents increasingly stand in for a specific person or character, in digital twins, social simulation and expert decision support. The characters the field builds for are canonical figures from widely circulated fiction, and each arrives with a gold-standard profile. Users also want niche characters, original creations and people they know, none of which come with a profile or with an identity the backbone can draw on. Hiding the name of a known character does not close this gap, since removing the identity also removes most of the value the profile carried. We propose Minstrel, which rebuilds a profile from free description and delivers it identity-free. It poses three profiling questions for each of eight dimensions drawn from personality psychology, converts the material each question retrieves into personality content, and assembles one conditioning text. Across four open-source actors and two cross-family judges, every Minstrel configuration beats a no-information control. Built from public comments alone, the profile reaches win shares of 0.570 and 0.610 while the actor identifies the character in 1.6% of trials. It also keeps the sixteen characters as distinct as their official profiles do. The code are available at https://anonymous.4open.science/r/Minstrel.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
role-playing agents, persona, character profile reconstruction, profiling, personality assessment, large language models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_AwyGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper works on a character personality reconstruction problem from unstructured descriptions in the role-playing setup. The problem assumes that no canonical profiles or identities are available in reconstruction, which poses additional challenges to the task.

As the main contribution, the author proposes a four-stage Minstrel frameowk that (1) specifies personality information needs through 24 questions across eight dimensions, (2) retrieves relevant evidence from available sources, (3) converts the evidence into personality cues or summaries, and (4) assembles an identity-free character profile. The paper also proposes an evaluation framework covering downstream role-playing utility, character distinctiveness, personality fidelity, and identity concealment.

The authors evaluate Minstrel using 8,673 public comments about 10 fictional characters. They adopt 4 open-source LLMs for construction and another 2 LLMs as judges. They compare 2 profile-constrcution strategies and 3 information sources to verify the feasibility of the Minstrel framework. In the experiments, comment-derived profiles achieve win rates of approximately 56–62% against a no-information control, while maintaining low character-identification rates. Additional analyses examine personality differentiation, agreement with official profiles, and comparisons with existing role-playing systems.

Critical strengths
  1. The authors provide a meaningful problem formulation. The no-canonical-profile assumption makes the problem closer to real-world use cases. The solution to the problem will be benenificial to both research and industry communities.

  2. The authors provide a systematic framework for personality evaluation. It considers utility, personality distinctiveness, fidelity, and identity concealment that are supported by psychological theories. When using LLM as judge, the authors also select multiple judges and add variations in the sample order to reduce bias.

  3. Empirical evidence shows that the proposed method Minstrel can improve personality reconstruction. The authors evaluate multiple information sources and profile-construction strategies across four open-source actor models. Comment-derived profiles achieve approximately 56–62% win rates against a no-information control while maintaining low character-identification rates. These findings provide evidence that useful personality information can be extracted from unstructured descriptions without explicitly revealing character identities.

Critical weaknesses
  1. The paper is difficult to follow due to overly abstract terminology, convoluted explanations, and excessive reliance on mathematical notation for relatively straightforward operations. In particular, Section 3 makes a simple retrieval-and-summarization pipeline appear more complicated than necessary. It would also work if the authors can use a concrete example to introduce those concepts.

  2. The paper misses several important aspects of impelmentation details and the experimental setup. For example:

  • What is an item in RoleAgentBench ? How do you apply 3 criteria to retain items?
  • What is a no-information control? What exact instructions or prompts are provided to the actor in this baseline?
  • What information is provided to the judges? What is the rubric?
  • What are the LLM sources? Are they the 4 open-source backbones? or the gpy-5.6-sol ?

Although the appendices provide some implementation details, these questions remain insufficiently addressed. Without a more transparent protocol, it is difficult to reproduce the results or assess potential evaluation biases.

  1. Minstrel primarily combines well-established techniques, including embedding-based retrieval, reranking, LLM-based personality extraction or summarization, and structured profile assembly. The fixed set of 24 questions across eight personality dimensions provides an organizational framework, but the paper does not clearly demonstrate a substantial technical innovation beyond integrating these existing components.
Questions for authors

In addition to the weakness,

  • What are the LLM sources? Are they the 4 open-source backbones? or the gpy-5.6-sol ?
  • How are Big Five scores generated? is table 3 averaging over samples and judges?
5 comments
AuthorsOpenReview ↗

Response to Reviewer AwyG - Part I

We thank Reviewer AwyG for recognizing the practical value of our problem formulation, our four-dimensional psychological evaluation framework, and our empirical findings.

1. Response to Weakness 1: Formal Notation in Section 3 and Adding a Concrete Running Example

Because this paper introduces a general design paradigm rather than a dataset-specific script, the lightweight mathematical notation in Section 3 (the information need set NN, dimension map dd, and stage operators A,C,G\mathcal{A}, \mathcal{C}, \mathcal{G}) serves an important structural purpose. It defines the formal interface signatures that allow different information sources (Principle P2) and conversion strategies (Principle P3) to be swapped without altering downstream stages, and it connects Section 3 directly to the evaluation metrics in Section 4 (W(c),ρf,cov(c),ε(c)W(c), \rho_f, \mathrm{cov}(c), \varepsilon(c)) and the constrained objective in Appendix B (Equation 10, Line 733). These signatures give future researchers a precise specification to follow when extending the paradigm.

At the same time, we fully agree with your suggestion that introducing these concepts alongside a concrete example makes Section 3 much easier to follow. In the revised manuscript, we will streamline the prose in Section 3 and add a concrete running example across Sections 3.2 to 3.5 (tracing how Question D5-Q2 for hp_mcgonagall retrieves raw comments mnm_n, converts them into de-identified personality cues or summaries unu_n, and assembles the final profile section csc_s).

2. Response to Weakness 2: Implementation Details and Experimental Setup

Below are the exact implementation and evaluation protocols (which we will add to Appendix C):

  1. What is an item in RoleAgentBench, and how are the 3 filtering criteria applied? Each open-ended question-answering item in RoleAgentBench consists of three parts: (i) the interlocutor's utterance addressed to the target character, (ii) the target character's name, and (iii) the script-derived reference response (shown only to the Judge, never to the Actor). Across the original 331 items, we applied three criteria sequentially:
    • knowledge_requirement != essential: excluded 118 items that required recalling unseen canonical plot events or franchise-specific terms absent from the visible prompt.
    • input_sufficiency == sufficient: excluded 118 items whose visible input alone could not determine the response direction.
    • profile_relevance != none: ensured that the item tests stable personality traits, retaining 95 items across 10 characters.
  2. What is a no-information control (∅\emptyset), and what exact prompt is provided to the Actor? In the no-information control (B-none), the Actor receives the same message as in the informed conditions, but the conditioning field is an empty string. The fixed system instruction is: "Role-play the target described below. The target's identity is deliberately withheld: do not guess it, do not state it, and do not name any specific real or fictional work. Do not mechanically list the supplied traits. Do not invent biographical facts that the supplied material does not support. Respond naturally to the situation in the target's expressive style and behavioural pattern." The user message is a JSON object with the fields identity (the word withheld), conditioning, context and query. Decoding uses model defaults with sampling, up to 512 new tokens.
  3. What information is provided to the Judges, and what is the rubric?
    • Input Information: In each pairwise comparison, the Judge receives the question put to the character, the reference answer from RoleAgentBench, and two candidate responses labelled CANDIDATE A and CANDIDATE B, with condition labels hidden. Every pair is judged independently in both display orders (AB and BA), and any pair whose verdict changes with order is recorded as undecided.
    • Scoring Rubric: The Judge decides which candidate is closer to the reference in (a) factual content and (b) speaking style, attitude and persona. The placeholder [TARGET] is deliberate masking and is ignored, length alone is not quality, and the Judge may answer tie.
  4. What are the LLM sources (Source = LLM)? Source = LLM (the LLM rows in Table 2) uses the external model gpt-5.6-sol (see Line 291 and the Profiler row in Appendix D Table 10), queried by the character's real name with the 24 profiling questions (prompt in Appendix C, Lines 814-821) to test eliciting a profile from an external LLM with prior knowledge of the character (such as a user's companion agent). By contrast, the 4 open-source models (Qwen2.5-7B/14B, Llama-3.1-8B, Mistral-7B) serve strictly as downstream Actors and never build profiles or judge responses.
AuthorsOpenReview ↗

Response to Reviewer AwyG - Part II

3. Response to Weakness 3: Technical Contribution vs. Minimal Step Implementations

We do not claim to have invented a new underlying retrieval algorithm or summarization model. As introduced in Lines 105-117, our core contributions lie in defining a new practical problem class (role-playing subjects without gold-standard profiles or pretrained identity memory) and proposing a general, psychology-grounded profile reconstruction design paradigm along with a four-level controlled evaluation protocol.

When validating this new paradigm in our experiments, we deliberately adopted the simplest and most direct implementation for every step of the paradigm. This design choice serves two key scientific purposes:

  • Eliminating confounding variables: It demonstrates that the significant downstream utility gains (Table 2), the 100% dimension coverage rate (Table 4), and the high character distinctiveness (Table 3) genuinely stem from the 8-dimension, 24-question identity-free reconstruction paradigm itself, rather than from specialized engineering tricks.
  • Providing a clean, transparent baseline for future research: As diagnosed in Section 4.4, third-party comments exhibit a 4.7% to 10.2% contradiction rate against official profiles. Our minimal paradigm implementation establishes a clean baseline pipeline upon which future work can specifically design consensus-weighting and conflict-resolution algorithms.

4. Responses to Questions for Authors

  • Question 1 (What are the LLM sources?): As detailed in Item 4 of our response to Weakness 2 above, Source = LLM uses gpt-5.6-sol queried by character name to answer the 24 profiling questions, whereas the 4 open-source backbones are used exclusively as downstream role-playing Actors.
  • Question 2 (How are Big Five scores generated, and what does Table 3 report?): Big Five scores are generated by the independent evaluator Judge B (Kimi-K2.6), which did not participate in building any profiles or generating any actor responses. For each of the 16 subjects and each type of rated material (Official profile, Minstrel profile, and aggregated Actor responses), Judge B rates the subject on a 9-point scale (1 to 9) across each of the five Big Five factors (3 independent repeated ratings per cell, yielding 16×3=4816 \times 3 = 48 rating observations per factor with a within-cell standard deviation of 0.63). Table 3 does not average across multiple judges. Instead, for each factor ff, Table 3 reports the Intraclass Correlation (ρf=σb,f2/(σb,f2+σw,f2)\rho_f = \sigma^2_{b,f} / (\sigma^2_{b,f} + \sigma^2_{w,f}), Equation 7) from a one-way ANOVA across subjects, which measures the proportion of total rating variance that lies between different subjects relative to repeated-rating noise.

Dear Reviewer AwyG,

We wanted to briefly follow up to let you know that all revisions promised in our response have now been incorporated into the updated PDF (marked in red):

  • Streamlined Section 3 & concrete running example: We removed the composite stage equation, kept only lightweight stage-level signatures (A,C,G\mathcal{A}, \mathcal{C}, \mathcal{G}), moved concrete operator settings to Section 4.1, updated Figure 2, and added the new Table 2 (Section 3.5) tracing one profiling question through all four stages with real data.
  • Complete experimental & judging protocols: Section 4.1 and Appendix C (Items and filtering, plus verbatim system prompts) now report the RoleAgentBench item structure, the sequential filtering counts (331→213→95331 \to 213 \to 95 items), the exact actor instructions and JSON format for the no-information control (∅\emptyset), and the verbatim pairwise judge prompt and rubric. Appendix D (Table 13) explicitly separates the Profiler (gpt-5.6-sol), the four open-source Actors, and the two cross-family Judges.
  • Clarified positioning: Sections 1 and 3 clarify that each stage uses its most direct instantiation on purpose so that the empirical gains isolate the problem formulation and psychological interface rather than operator engineering.

We hope these updates resolve your concerns regarding readability and reproducibility, and we remain happy to address any further questions.

Reviewer_6znEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reconstruct character profiles from scattered descriptions. Minstrel uses 24 questions across eight personality dimensions to gather material, turn it into personality cues or summaries, and assemble a profile with identifying details removed. The experiments compare three information sources: public comments, official profiles, and an external LLM. Four actor models and two judges are used to evaluate the results. In the pooled comparisons, reconstructed profiles outperform a control that provides no character information.

Critical strengths
  1. The research question has practical value. Building useful role-playing profiles from scattered descriptions is a meaningful task for people or characters without ready-made profiles. The comment corpus and processing pipeline could also support further research.

  2. The paper is generally easy to follow, and Figure 2 clearly explains the pipeline. The comparisons across information sources and processing strategies, along with swapped response order, judges from different model families, and identity-recognition checks, provide useful checks on the results.

Critical weaknesses
  1. The added value of the full pipeline over simpler profile-building methods remains unclear.

(a) The pipeline offers a reasonable way to organize scattered descriptions. I would appreciate clearer evidence of its benefit over simpler approaches using the same material. Table 2 shows that the resulting profiles help compared with providing no character information, but leaves open how much the proposed structure and processing steps contribute. The Summary variant retains the same question-based retrieval and eight-dimension assembly, so it does not directly address this question. The personality-augmentation approach in "Rethinking Role-Playing Evaluation: Anonymous Benchmarking and A Systematic Study of Personality Effects" (Ji-Lun Peng and Yun-Nung Chen, SIGDIAL 2026) provides a relevant reference for an alternative personality structure.

(b) Identity removal goes beyond deleting names: it also rewrites plot events, abilities, and relationships. These details may carry information important to the character. Appendix A shows that deeper identity removal is accompanied by lower performance, but does not fully separate the reduced benefit of character recognition from the loss of useful character information. A clearer check on what information is preserved would help explain the reconstructed profiles' performance.

  1. The evaluation scope, reference profiles, and some statistical interpretations limit the reliability and scope of the conclusions.

(a) The role-playing evaluation covers only 95 filtered items over ten well-known characters. Filtering is reasonable for focusing on personality, but the exclusions, selection criteria, and human validation need further explanation. Pooled win rates may also be influenced by a few characters. Per-character results and an analysis that accounts for correlations among responses for the same character would help. Anonymized familiar characters are still a proxy for unfamiliar subjects, and a low recognition rate does not rule out the use of relevant prior knowledge.

(b) Official profiles serve as both an input source and a reference for fidelity. However, the main text does not clearly explain their source or construction, or provide full examples of actual profiles before and after identity removal. This makes the quality of the reference difficult to assess.

(c) Section 4.6 and Table 5 describe the masked official-profile condition and the comment-reconstruction condition on the strongest backbone as equivalent to PersonaForge. Their win-share intervals, [43.8%, 50.4%] and [47.7%, 60.7%], include 50%. These intervals indicate that the current test did not detect a clear difference; they do not by themselves establish equivalence. The paper does not report an equivalence margin or a corresponding test, so the claim of matching an existing system would benefit from narrower wording.

Questions for authors
  1. Could you share the rationale for choosing eight dimensions and three questions per dimension? A comparison with a one-pass summary or an alternative personality structure, using comparable inputs and profile lengths and the same identity-removal rules, would help clarify what the proposed design adds.

  2. Could you clarify the statistical basis for the equivalence claims in Sections 4.2 and 4.6? Was an equivalence margin specified and tested, or should these results be described as comparisons in which no difference was detected?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6znE - Part I

We thank Reviewer 6znE for the constructive review and insightful suggestions on ablation analysis, evaluation details, and statistical reporting.

1. Response to Weakness 1(a): Paradigm Positioning and Why 8 Dimensions x 3 Questions Are Needed

We first clarify the positioning of our work. This manuscript focuses on problem definition and a general design paradigm, rather than proposing a single-source RAG or text-summarization algorithm. Because this design paradigm requires strong compatibility across settings, its structure serves requirements beyond static document compression:

  1. Why raw sources require question-driven acquisition rather than a one-pass summary: Minstrel is a basic design paradigm rather than a specific RAG method, so it must accommodate the full process from information acquisition to personality reconstruction. First, we must filter relevant personality content out of noisy comment corpora or other raw sources. We cannot simply feed hundreds or thousands of unfiltered comments directly into a one-pass summary (where over 40% recount plot events or meta-discussions, Appendix E), nor can we expect a human interviewee to articulate comprehensive personality information without specific profiling questions (Section 1, Line 91, Principle P2). Asking a human user for a one-pass summary without specific prompts shifts the profiling burden back onto the user, collapsing into manual persona customization (Table 1, Row 3).
  2. Theoretical grounding of the 8 dimensions x 3 questions: Our choice of 8 dimensions x 3 questions comes from classical theories in personality psychology, mapping directly onto the three-level architecture of dispositional traits (Stable tendencies), characteristic adaptations (Motives and goals, Values and moral priorities, Cognitive and appraisal style, Affect and coping, and Interpersonal patterns), and narrative identity and conditional situations (Self and narrative identity and Situation and expression). When designing specialized methods in future studies, we will carefully analyze the necessity of each component. In our experiments here, every step in the paradigm adopts a direct and simple implementation, which aligns with your perspective. Even when Stage 3 uses simple Direct Summarization under our 24-question acquisition and 8-dimension assembly, it achieves 100% dimension coverage (Table 4) and high distinctiveness (Table 3). Furthermore, we appreciate your pointer to Peng and Chen (SIGDIAL 2026), which has high reference value for our study. We will discuss this work in Related Work regarding how it uses a specific personality structure for persona augmentation, and contrast it with Minstrel's focus on reconstructing profiles from scratch from scattered descriptions via the 8-dimension, 24-question interface.

2. Response to Weakness 1(b): Separating Loss of Recognition from Loss of Character Information

In our four-level masking ladder (L0 -> L1 -> L2 -> L3), rewriting plot events, abilities, and relationships occurs only at L2 -> L3. From L0 to L2 (zero rewriting of events, abilities, or relationships, replacing only proper nouns via a lexicon), as character recognition falls from 90.6% to 37.5%, win share already drops from 0.769 to 0.697 (-7.2 percentage points, or 50.3% of the total 14.3-point drop).

To test whether the remaining 7.1-point drop from L2 to L3 (0.697 -> 0.626) comes from rewriting plot, ability, and relationship details or from recognition falling further from 37.5% to 6.2%, we stratified all 40 (Actor x Character) cells into three groups by whether their character-recognition status changed during L2 -> L3:

Stratum (L2 -> L3 Recognition)CellsL2 Win Share (No rewrite)L3 Win Share (Deep rewrite)Δ\DeltaCore Conclusion
1: Recognized (1 -> 1)40.788 [0.697, 0.857] (78/99)0.777 [0.682, 0.849] (73/94)-1.1%Holding recognition at 100%, rewriting causes almost zero loss.
2: Unrecognized (0 -> 0)170.658 [0.603, 0.709] (202/307)0.621 [0.565, 0.673] (193/311)-3.7%Holding recognition at 0%, rewriting causes only a minor change.
3: Collapsed (1 -> 0)190.703 [0.659, 0.744] (308/438)0.599 [0.552, 0.643] (264/441)-10.4%The drop is concentrated in the 19 cells that lose recognition.

Moreover, our dimension audit in Table 4 (Summary x Profile vs. unmasked L0 Official Profile across 128 cells) shows that after full identity, event, and relationship removal, 100% (124/124) of dimensions present in the original profile are preserved (75.8% Full + 21.1% Partial, 0.0% None, 0.0% Contradiction). We will add this analysis to Appendix A.

AuthorsOpenReview ↗

Response to Reviewer 6znE - Part II

3. Response to Weakness 2(a): Subject Selection, Per-Character Analysis, and Statistical Power

Thank you for raising these points, which we address in turn below. In terms of scale, the number of characters used in our experiments is indeed compact. However, due to real-world data availability, only these subjects remained after filtering through four strict requirements:

  1. Cross-benchmark alignment: Each character must possess both an official gold-standard profile (Official Profile) and paired script-derived test items in established role-playing benchmarks (RoleAgentBench or CharacterEval).
  2. Public comment availability and sufficiency on compliant platforms: Across the 6 public platforms permitting compliant collection, each character must have a sufficient volume of authentic, non-synthetic third-party comments (we collected 8,673 comments from 6,569 distinct authors, averaging ~540 comments per character). Many minor or less-discussed characters on compliant platforms simply do not have enough comment density to support retrieval across 24 profiling questions, especially for Chinese characters where high-quality long comments are even harder to obtain.
  3. Bilingual and iconicity balance: To avoid single-language bias, we deliberately balanced English and Chinese characters (10 English + 6 Chinese) as well as different popularity tiers.
  4. External independent psychometric scale availability (used for the external validation in Appendix E): Each character also needed independent MBTI, Enneagram, and Big Five community voting data on Personality Database (PDB) for external validity cross-checking.

Our profile-quality evaluations (Table 3 distinctiveness and Table 4 dimension audit) cover all 16 bilingual characters. Downstream question-answering reports the 10 English characters (95 items filtered from 331 by excluding 118 canon-knowledge items and 118 context-underdetermined items) because the benchmark hosting the 6 Chinese characters failed our positive-control instrument validity gate, where its reward model scored the gold-standard profile below the no-information baseline. Out of scientific rigor, we did not mix those invalid instrument scores into the main table and reported this fully in the first paragraph of Limitations (Appendix F, Lines 972-982).

Furthermore, to ensure that our results reliably support our conclusions and are not driven by a few characters, we conducted both character-level analyses and small-sample significance tests:

  • Per-character analysis: Giving all 10 characters equal weight (1/10 each, removing item-count differences across characters) yields unweighted character-level mean win shares of 0.574 (Judge A) and 0.631 (Judge B) for Extractor x Comments, and 0.565 (Judge A) and 0.625 (Judge B) for Summary x Comments, all slightly higher than the pooled win shares (0.570 / 0.610 and 0.563 / 0.622). Across the 10 characters, Minstrel's per-character win shares also closely track the ground-truth De-identified Official Profile per-character win shares with Pearson r=0.942r = 0.942 (p=4.6×10−5p = 4.6 \times 10^{-5} for Extractor) and r=0.927r = 0.927 (p=1.1×10−4p = 1.1 \times 10^{-4} for Summary).
  • Downstream utility significance (Table 2): 95 items x 4 Actors x 3 replicates x 2 display orders yield 49,305 generated responses and 113,971 pairwise verdicts. All 12 Wilson 95% CIs across the 6 configurations and 2 cross-family judges lie strictly above 0.50, and all Holm-corrected exact sign tests are statistically significant.
  • Character-level significance (Table 3 & Section 4.4): Even when taking the character as the unit of analysis (N=16N=16 and N=10N=10), the one-way ANOVA Intraclass Correlations across all Big Five dimensions (ICC 0.86-0.97) are highly significant (p<1.4×10−5p < 1.4 \times 10^{-5}), and the correlation between the behavioral pairwise distance matrix and the official profiles is also significant (r=0.541,p=2.5×10−5r = 0.541, p = 2.5 \times 10^{-5}).

These results confirm that our experiments already yield highly significant statistical conclusions sufficient to support our claims. We also transparently disclosed and discussed these scope limitations in Appendix F (Limitations, Lines 972-999), and as noted in our response to Reviewer MeRq, we will expand our benchmark in future work.

AuthorsOpenReview ↗

Response to Reviewer 6znE - Part III

4. Response to Weakness 2(b): Provenance of Official Profiles and End-to-End Data Journey Examples

1. Provenance of Official Profiles and Comments: All Official Profiles come directly from publicly released benchmarks in prior studies rather than being written by us: the 10 English Official Profiles come verbatim from the role_summary files of RoleAgentBench (Liu et al., 2024), and the 6 Chinese Official Profiles come verbatim from CharacterEval (Tu et al., 2024). Public comments were compliantly collected from six online discussion communities (Appendix D, Table 9). As discussed in Section 4.4 and Limitations, raw public comments naturally contain viewer disagreements and contradictions (4.7% to 10.2% contradiction rate), serving as a realistic testbed for evaluating reconstruction from noisy descriptions.

2. End-to-End Data Journey (hp_mcgonagall, Dimension 5 Question 2: "How does this person's outward expression differ from their apparent inner emotion?"):

  • Journey A: Comment-Sourced Reconstruction (S-a)
    • Retrieved Raw Comments (mnm_n): (StackExchange) "She's competent, strict, and an all-around ass-kicker with a bit of a soft spot for her students that she tries to hide..." / (YouTube) "...She was still fighting inside with her feelings towards one she called a colleague/friend for so many years..."
    • Converted Content (unu_n): Under Extractor [Strong]: "[TARGET] presents a strict, highly competent exterior while concealing tenderness and protectiveness toward those in her care." Under Summary: "[TARGET] outwardly appears strict, highly competent, composed, and decisive... Beneath that firm exterior, she seems deeply protective of those in her care and reluctant to seriously harm a longtime colleague..."
    • Assembled Profile Excerpt (csc_s, section "Affect and Coping"): Extractor: "Conflict with a longstanding associate produces visible restraint despite inner anger, attachment, and uncertainty." Summary: "Their controlled exterior masks protectiveness, affection, vulnerability, and possible guilt."
  • Journey B: Official Profile Before/After Identity Removal (L0 -> L3) and Pipeline Reconstruction (S-b)
    • Original Official Profile (L0-raw, RoleAgentBench verbatim): "Minerva McGonagall is a stern but fair professor at Hogwarts School of Witchcraft and Wizardry, known for her sharp wit, strict demeanor, and unexpected kindness. The Deputy Headmistress, head of Gryffindor House, and Transfiguration teacher, she is respected for her strong sense of duty..."
    • Deeply De-identified Official Profile (L3-deep): "[TARGET] is a stern but fair senior teacher at a residential school, known for her sharp wit, her strict bearing, and a kindness that surfaces only when it is not expected. As deputy head of the institution, head of the student group associated with courage, and the teacher of one of its most demanding technical disciplines, she is respected for a powerful sense of duty..."
    • Minstrel Reconstruction (C-a_S-b on D5-Q2): [Strong] "[TARGET] masks substantial kindness, concern, and protectiveness behind a stern, strict, and authoritative exterior."

5. Response to Weakness 2(c): Formal Equivalence (TOST) and Non-Inferiority Analysis

Our goal in Sections 4.2 and 4.6 was to test whether these conditions exhibit a performance gap relative to the reference baselines. Because their actual performance is comparable (for example, on the strongest backbone, Minstrel comment reconstruction achieves a 0.543 win share against PersonaForge, slightly above the 0.500 parity line), two-sided difference tests naturally yield 95% CIs that include 0.50. Following your suggestion, we will supplement Sections 4.2 and 4.6 with formal Exact Binomial Two One-Sided Tests (TOST) for equivalence (H0:∣W−0.50∣≥ΔH_0: |W - 0.50| \ge \Delta) and One-Sided Non-Inferiority Tests (H0:W≤0.50−ΔH_0: W \le 0.50 - \Delta). All reported p-values below are smaller than 0.05.

ComparisonDecided (k/n)Share90% Wilson CINon-inferiority (Δ=0.05\Delta = 0.05)Bidirectional TOST equivalence
Sec 4.2: Extractor x Profile vs. De-id Profile363 / 7400.491[0.460, 0.521]p=0.0148p = 0.0148Δ=±0.05\Delta = \pm 0.05: p=0.0148p = 0.0148, Δ=±0.06\Delta = \pm 0.06: p=0.0032p = 0.0032
Sec 4.2: Summary x Profile vs. De-id Profile391 / 7480.523[0.493, 0.553]p<0.0001p < 0.0001Δ=±0.06\Delta = \pm 0.06: p=0.0221p = 0.0221, Δ=±0.08\Delta = \pm 0.08: p=0.0009p = 0.0009
Sec 4.6: Masked Official Profile vs. PersonaForge407 / 8640.471[0.443, 0.499]p=0.0357p = 0.0357 (at Δ=0.06\Delta = 0.06)Δ=±0.06\Delta = \pm 0.06: p=0.0357p = 0.0357, Δ=±0.08\Delta = \pm 0.08: p=0.0014p = 0.0014
Sec 4.6: Minstrel (Comments, Qwen-14B) vs. PersonaForge121 / 2230.543[0.488, 0.597]p=0.0034p = 0.0034Δ=±0.10\Delta = \pm 0.10: p=0.0470p = 0.0470
Reviewer_dwSHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary
  • This paper proposes a system that makes an LLM embody a persona based on a set of parameters that are passed in
  • The paper focuses on simulating persons of which there is no gold-standard profile. In contrast to existing methods that focus on embodying personas of existing people, Minstrel simulates personas that do not exist in real life.
  • Rather than reconstructing an existing profile, Minstrel creates a new profile from a set of descriptions. The authors call this task profile reconstruction from descriptions.
  • Minstrel asks 24 profiling questions over 8 personality dimensions and outputs a conditioning text. The conditioning text has personality information about the persona that is being role-played and is pass into an LLM to play the role.
Critical strengths
  • The paper's motivation and field are interesting. Role-playing agents are useful for a wide range of tasks beyond mere entertainment.
  • The paper's connection to classical psychology is also interesting
Critical weaknesses
  • The paper is confusing to read.
  • This sentence in the introduction is confusing:

Removing identity costs 14.3 points of win share, which is most of the margin the profile carried. Actor-side recognition falls from 90.6% to 6.2% across the four levels, and the win share falls from 0.769 to 0.626 in step with it.

  • The "win share" is not defined before its use in this sentence.
  • The "backbone"" is brought up multiple times in the paper but is not defined.
  • This section is confusing

(a) A user turns down familiar characters, each summarised by the headline attributes of its profile. The new character she settles on combines one attribute from each, and no existing profile describes it. (b) Specification, carried out once for a population.

  • It is unclear to me what it means to "turn down a character"

Hiding the name is also the usual workaround, so we measured what it costs.

  • This sentence is confusing

Self and narrative identity is the level of the life narrative, and situation and expression captures the if-then signatures through which a stable personality shows differently across situations.

  • This sentence is confusing

Under situation and expression, one question asks which situations are exceptions to the person’s general tendencies, which is the if-then signature stated as a question

Questions for authors
  • I would love to understand better what the problem is with the "usual workaround"? The problem the paper tries to solve is to create a profile based on a list of descriptions. How does "the usual workaround" help with this? My understanding of the usual workaround is that you pass in a character with its descriptions and you strip the name. How does that help role-play a character from descriptions only? I imagine this could be a useful eval, though?
6 comments
AuthorsOpenReview ↗

Response to Reviewer dwSH - Part I

We thank Reviewer dwSH for giving our paper the highest possible soundness score (Soundness = 3) and for noting that our motivation and connection to classical psychology are interesting. Below we clarify two misunderstandings in the Paper Summary, provide the revised sentences for your clarity concerns, and answer your question.

1. Clarifying Two Points in the Paper Summary

  • Target subjects: Rather than simulating personas that do not exist in real life, our primary target is real-world ordinary people (such as a parent, friend, or classmate, Section 1, Lines 50-52) and non-canonical characters that lack a gold-standard profile. In our experiments, we use four-level de-identified canonical characters as a controlled research vehicle. This design protects real-person privacy while providing objective ground-truth references (Official Profiles) and standardized test items (RoleAgentBench) to rigorously verify reconstruction fidelity.
  • Paper positioning: Our manuscript is positioned as problem definition + basic design paradigm + empirical analysis, rather than proposing a specific black-box engineering system. Following Reviewer MeRq's feedback and your review, we have separated the general paradigm presentation from specific experimental conditions so that the paper's positioning and prose are much clearer and easier to read.

2. Point-by-Point Sentence Revisions for Your Clarity Concerns

Since all items in your Critical Weaknesses concern unclear sentence phrasing or missing term definitions, we address them together here and list the revised sentences. Briefly, win share means win rate, referring here to the pairwise win rate of a role-playing model conditioned on a de-identified profile against a no-information control when answering test items. Backbone refers to the underlying language model used for role-playing. Turns down means choosing not to select or declining: Figure 2(a) illustrates a user visiting a character shop who does not want any of the pre-made canonical characters on the shelf and therefore rejects them. We have revised the six passages you noted as follows:

  • Item 1 (Section 1, Lines 80-83): "We measure the downstream benefit of a profile by its win share against a no-information control, defined as the proportion of decided pairwise comparisons in which a judge prefers the profile-conditioned response over an unconditioned response. Across the four masking levels, the actor model's ability to recognize the character falls from 90.6% to 6.2%, and its win share falls in step with it from 0.769 to 0.626 (a loss of 14.3 percentage points, erasing most of the advantage the official profile originally provided over the 0.500 no-information baseline)."
  • Item 2 (Abstract Line 16 & Section 1 Line 42): (Abstract) "...none of which come with a profile or with an identity the underlying language model (backbone) can draw on." (Section 1) "...and an identity the underlying LLM acting as the role-player (hereafter, the backbone) may have encountered during pretraining."
  • Item 3 (Figure 2 Caption, Lines 182-185): "(a) Familiar canonical characters each come with a pre-existing profile, but none matches the target person the user actually wants to simulate (such as an ordinary person in real life or a custom character), for whom only scattered descriptions exist. (b) Specification of profiling questions, defined once and shared across all subjects."
  • Item 4 (Section 1, Line 79): "In prior evaluations of unknown-character role-playing (Table 1, Row 4), simply hiding a famous character's name in their existing official profile is the standard workaround used to simulate an unfamiliar subject. To test whether this workaround truly reflects the difficulty of playing an unknown person, we took the official profiles of sixteen canonical characters and progressively stripped identifying cues across four levels while holding personality content fixed."
  • Item 5 (Section 3.2, Lines 215-217): "The seventh dimension, Self and narrative identity, captures how a person interprets formative life experiences and maintains a coherent self-image. The eighth dimension, Situation and expression, captures context-dependent behavioral patterns, known in personality psychology as if-then behavioral signatures (i.e., if placed in situation X, the person reacts in way Y)."
  • Item 6 (Section 3.2, Lines 220-221): "For instance, under the Situation and expression dimension, one profiling question asks under which specific circumstances the person departs from their usual behavior, directly phrasing these conditional (if-then) behavioral patterns as a concrete question."
AuthorsOpenReview ↗

Response to Reviewer dwSH - Part II

3. Response to Question on "The Usual Workaround"

Your intuition at the very end of your question ("I imagine this could be a useful eval, though?") is spot-on, as it indeed serves as an evaluation control condition in our study. Building on our positioning clarification above, this paper primarily formulates a new problem class and establishes a feasible research paradigm, rather than proposing a narrow engineering method that only builds characters from a static text list. Our goal is to study how to construct non-canonical personas that have no gold-standard profile and are much closer to ordinary people around us. The model does not know their names, and even if given a name, it cannot use that name to anchor the person's personality traits. Extracting personality cues from third-party comments is one information-acquisition mode used in our experiments. As Reviewer MeRq noted (and as stated in Lines 91, 225, and 233 of our manuscript), personality information in Minstrel can be acquired not only from third-party comments, but also by querying an external LLM or having a human user answer the profiling questions directly.

Due to benchmark and reference availability constraints, we used canonical characters as the experimental vehicle in Section 4. Consequently, our experimental design had to eliminate the confounding influence of known character identities on role-playing performance. The phrase "the usual workaround" does not refer to other persona-construction baselines. Instead, it refers to a common operation in prior studies that analyze the impact of character identity by simply deleting the character's name from an existing profile (Table 1, Row 4). However, our experiments show that merely deleting the character's name is far from sufficient to hide the character's identity. For example, if an LLM or a familiar reader sees the description "the best bard in Mondstadt", even without the name "Venti" appearing anywhere, it immediately identifies the character and draws on pretrained memory. We therefore designed a stricter four-level de-identification ladder. Thus, "the usual workaround" refers to an experimental control condition rather than a competing reconstruction method, which also resolves the follow-up questions in your comment.

Regarding this transition, an earlier draft of our manuscript included the following passage right before Line 79 ("Hiding the name is also the usual workaround, so we measured what it costs."):

"Both gaps can be probed with one manipulation, because a canonical character can be stripped of exactly what an ordinary subject never had. We therefore took gold-standard profiles and removed the identity from them in four steps... Stripping a profile in this way is also what an unknown-character condition usually means in practice."

We had trimmed those sentences for space under the assumption that the surrounding text and Table 1 Row 4 provided sufficient context. In the revised manuscript, we have restored this explicit explanation and reviewed the entire text to ensure smooth readability.

Reviewer_dwSHOpenReview ↗

Response to Authors

Thank you authors for your careful responses.

However, I still find the draft not well-written, and this is compounded by the dense mathematical notation that Reviewer AwyG mentioned. I stand by my statement that this draft is too challenging to read and I keep my score.

Moreover, the technical contribution here seems slim. Am I right to understand that the pipeline you are proposing only involves prompt and retrieval? The proposed pipeline should have a thorough ablation to understand if all the components of this 24-question 4-stage pipeline are equally important.

Reviewer_MeRqOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper is the first to focus on the ordinary-person role-playing task under the condition of "having no gold-standard settings and being unknown during pre-training." The manuscript first analyzes and verifies, through anonymization experiments, the massive implicit gains of character names in role-playing. Afterwards, the authors propose Minstrel, a design pipeline that reconstructs characters relying on third-party comments. Drawing on personality psychology theories, Minstrel constructs 8 major psychological dimensions and 24 corresponding profiling questions. Based on these 24 questions, the authors retrieve relevant answers from collected online comments and extract personalities from the corresponding comments. It should be noted that in the experiments of this paper, the authors used multiple conditions such as personality cue extraction and direct summarization. Finally, Minstrel organizes this personality content into a complete, identity-free profile to guide the LLM in completing the role-playing task. The authors validate the effectiveness and distinctiveness of the profiles constructed by Minstrel through extensive experiments. Under certain conditions, the profiles constructed by Minstrel can even match or outperform gold-standard profiles and existing role-playing methods.

Critical strengths

According to the guidelines, two critical strengths supporting acceptance are given below:

  1. This manuscript breaks through an obvious blind spot in the existing AI role-playing field and defines a new task for ordinary-figure role-playing that is closer to real-world scenarios. At the same time, the authors introduce relevant theories from psychology and cognitive science to design a basic design pipeline, Minstrel. Without relying on ready-made gold-standard settings or having been seen during model pre-training, it constructs characters and completes the role-playing task solely by relying on fragmented descriptions to answer profiling questions. It is worth mentioning that as generative models exhibit more and more anthropomorphic behaviors, I believe we should accept more research work related to Theory of Mind to adapt to the development of this topic.
  2. In addition to defining the task and the basic execution pipeline, this manuscript also conducts extensive rigorous empirical research and makes strong theoretical contributions. On the one hand, through a four-level anonymization experiment, the authors verify the false prosperity in existing role-playing where large models "cheat" by relying on names. This further strengthens the conclusions obtained from some previous studies that only deleted character names, and quantifies the implicit gains brought by character names. On the other hand, this manuscript verifies the possibility of building high-quality role-playing agents relying solely on third-party comments. Furthermore, the authors not only verify the core role-playing effects, but also design rigorous significance tests and distinctiveness experiments, introducing professional scales for evaluation to ensure the reliability of the research conclusions and the Minstrel pipeline.

In addition, these strengths are also provided for the Area Chair's reference when making decisions:

  1. As a basic design pipeline, Minstrel has strong compatibility. It not only accommodates different experimental conditions in this paper's experiments (such as cue extraction, summarization, etc.), but also provides strong inspiration for future method design research in this direction. In addition, it does not restrict the source of information; as long as these 24 questions can be answered, whether it is messy netizen comments, gold-standard settings, external large models, or online answers from real human users, they can all be seamlessly integrated.
  2. Beyond the main role-playing effect and distinctiveness experiments, the authors also conducted a detailed dimensional analysis, showing that Minstrel achieved a 100% coverage rate across the 8 personality dimensions. At the same time, they discovered the key issue of contradictory content in third-party comments, which also points out a research direction for handling consensus issues in comments in the future.
  3. Experimental results show that Minstrel's approach of reconstructing characters using third-party comments is very effective. Under the condition of completely erasing identities and using only netizen comments, Minstrel's performance on the strongest backbone model tied with the existing system PersonaForge, which holds both "gold-standard profiles + character names", and overwhelmingly defeated the CoSER system with a win rate of over 0.82.
  4. It introduces theoretical support from psychology and cognitive science, integrating the Big Five personality traits, characteristic adaptations theory, and cognitive-affective system theory to extract affective attributes covering traits, motivations, values, etc., providing theoretical validation for the mental alignment between humans and intelligence.
Critical weaknesses

No critical weaknesses sufficient to support rejection were found. Some other points for the authors' reference in making revisions can be found in the "Questions For Authors" section.

Questions for authors
  1. True no-name minor characters were not collected; having them would provide more direct support for the content of this study. Although the current manuscript has already conducted rigorous anonymization experiments and controlled experiments, the chosen characters themselves still do not directly fit the setting of minor characters. However, that would mean requiring stricter IRB review, which can perhaps be further supplemented in other future studies. Do the authors have plans in this regard in the future?
  2. When describing the experimental methods, especially in Figure 2, the packaging of the feature extractor is somewhat excessive. Looking at it as a whole, this is a good empirical analysis paper, and whether it is personality cue extraction or summarization, both are just experimental conditions of Minstrel. Over-packaging can easily lead to reader confusion regarding the positioning of the paper. I understand that the authors hope to align with the mainstream direction of design science research in CS, especially in the AI field. However, I do not think this is necessary. It is quite enough for this manuscript to fulfill the mission of "problem definition + empirical analysis." Overall, the packaging of the method in Figure 2, especially the personality cue feature extractor, gives a sense of being superfluous (gilding the lily). Please respond on how you will revise this point, and be as specific as possible.
  3. Considering that design science remains a mainstream research trend, I need the authors to clarify: Do the authors position Minstrel as a basic design pipeline as I understand it, or as a specific method?

If these questions receive reasonable responses from the authors, I will consider raising the score appropriately.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MeRq

We thank Reviewer MeRq for the thorough assessment, the strong support for our problem formulation and empirical analysis, and the constructive guidance on clarifying the positioning of our work. Below we address each of your three questions in order.

1. Response to Question 1 (Using canonical characters as a research vehicle and future plans for ordinary people)

Your observation goes to the heart of our experimental design. Modeling real ordinary people directly requires strict IRB review and informed consent from both the subjects and the third-party observers describing them. Beyond ethical review, a practical challenge arises because our work is the first to formulate the ordinary-person role-playing task under conditions with no gold-standard profile and no pretraining identity prior. The field currently lacks public datasets, evaluation frameworks, and reference baselines for real ordinary people.

To obtain reliable evaluation resources and verify the fidelity of reconstructed profiles against a ground-truth reference, we used canonical characters as a controlled research vehicle. At the same time, we conducted our four-level identity removal experiments to measure role-playing performance after stripping away character names and identifying anchors. At Level L3, where actor-side recognition drops to 6.2%, the actor model faces a subject it cannot recognize, which mirrors the condition of playing an ordinary person without pretraining anchors.

Our answer regarding future work on real ordinary people is a firm yes. As a foundational study that establishes the problem boundaries, basic paradigm, and evaluation protocol for this direction, this manuscript lays the groundwork for our ongoing research program. In our next studies, we will recruit human participants under IRB approval to collect real-world ordinary-person descriptions and build a dedicated benchmark. We will also explore specialized persona reconstruction algorithms (such as consensus weighting for conflicting descriptions), alongside planned studies on situational adaptability and group-level personality differences.

2. Response to Question 2 (Revising Figure 2 and aligning the narrative with problem definition and empirical analysis)

We agree with your assessment and appreciate this advice. Our primary goal in this paper is to define this overlooked problem class, establish a basic resolution paradigm, and validate both the problem and the paradigm through empirical analysis. Mixing method-paper presentation habits into Section 3 and Figure 2 obscured this positioning.

We will make three concrete revisions in the updated manuscript:

  • Redrawing Figure 2: We will remove the visual emphasis that makes the personality cue extractor look like a standalone algorithmic black box. Instead, Figure 2 will present the four-stage paradigm workflow and display Cue Extraction and Direct Summarization side by side as interchangeable experimental conditions within Stage 3.
  • Refining Section 3: We will present Section 3 in general paradigm terms that highlight the four stages and five design principles, moving specific implementation choices to the experimental setup in Section 4.
  • Sharpening the Introduction and Conclusion: We will revise the framing sentences in Sections 1 and 5 so that readers immediately see the paper's core mission as problem definition + basic design pipeline/paradigm + multi-condition empirical analysis.

3. Response to Question 3 (Positioning Minstrel as a basic design pipeline vs. a specific method)

We fully agree with your judgment. Minstrel is positioned as a general basic design pipeline rather than a specific feature-extraction method. As you summarized in the Strengths section, Minstrel accommodates diverse information sources (noisy netizen comments, official profiles, external LLMs, or live human answers) and different processing conditions because it represents a design paradigm shift from relying on pretrained identity memory or pre-written profiles to reconstructing identity-free profiles from scattered descriptions via psychological theory.

The four-stage Minstrel basic design pipeline is the operational vehicle that grounds this paradigm and enables systematic empirical analysis. Under this positioning, both the personality cue extractor and direct summarization are experimental conditions at Stage 3 used to test the pipeline across different granularities of information conversion. Revising Figure 2 and Section 3 as described above will make this positioning unmistakable.

Thanks to the authors' response and the revised manuscript. Meanwhile, I am thrilled to come across this interesting and meaningful research work and feature rigorous experimental designs. The main issue with the previous version was that the position of Minstrel wasn't quite clear. The authors' response and the revised version have addressed my concerns. Thus, I will improve my scores. In addition, there are a minor issues the author should address. Although I would like the authors to further clarify the paper's position, there is no need to state it so explicitly: "Minstrel is a pipeline".

Thank you for the follow-up and for reading the revised manuscript. We agree that the positioning does not need to be stated so explicitly. We have toned down these statements in the updated PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.