- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper works on a character personality reconstruction problem from unstructured descriptions in the role-playing setup. The problem assumes that no canonical profiles or identities are available in reconstruction, which poses additional challenges to the task.
As the main contribution, the author proposes a four-stage Minstrel frameowk that (1) specifies personality information needs through 24 questions across eight dimensions, (2) retrieves relevant evidence from available sources, (3) converts the evidence into personality cues or summaries, and (4) assembles an identity-free character profile. The paper also proposes an evaluation framework covering downstream role-playing utility, character distinctiveness, personality fidelity, and identity concealment.
The authors evaluate Minstrel using 8,673 public comments about 10 fictional characters. They adopt 4 open-source LLMs for construction and another 2 LLMs as judges. They compare 2 profile-constrcution strategies and 3 information sources to verify the feasibility of the Minstrel framework. In the experiments, comment-derived profiles achieve win rates of approximately 56–62% against a no-information control, while maintaining low character-identification rates. Additional analyses examine personality differentiation, agreement with official profiles, and comparisons with existing role-playing systems.
Critical strengths
-
The authors provide a meaningful problem formulation. The no-canonical-profile assumption makes the problem closer to real-world use cases. The solution to the problem will be benenificial to both research and industry communities.
-
The authors provide a systematic framework for personality evaluation. It considers utility, personality distinctiveness, fidelity, and identity concealment that are supported by psychological theories. When using LLM as judge, the authors also select multiple judges and add variations in the sample order to reduce bias.
-
Empirical evidence shows that the proposed method Minstrel can improve personality reconstruction. The authors evaluate multiple information sources and profile-construction strategies across four open-source actor models. Comment-derived profiles achieve approximately 56–62% win rates against a no-information control while maintaining low character-identification rates. These findings provide evidence that useful personality information can be extracted from unstructured descriptions without explicitly revealing character identities.
Critical weaknesses
-
The paper is difficult to follow due to overly abstract terminology, convoluted explanations, and excessive reliance on mathematical notation for relatively straightforward operations. In particular, Section 3 makes a simple retrieval-and-summarization pipeline appear more complicated than necessary. It would also work if the authors can use a concrete example to introduce those concepts.
-
The paper misses several important aspects of impelmentation details and the experimental setup. For example:
- What is an item in RoleAgentBench ? How do you apply 3 criteria to retain items?
- What is a no-information control? What exact instructions or prompts are provided to the actor in this baseline?
- What information is provided to the judges? What is the rubric?
- What are the LLM sources? Are they the 4 open-source backbones? or the gpy-5.6-sol ?
Although the appendices provide some implementation details, these questions remain insufficiently addressed. Without a more transparent protocol, it is difficult to reproduce the results or assess potential evaluation biases.
- Minstrel primarily combines well-established techniques, including embedding-based retrieval, reranking, LLM-based personality extraction or summarization, and structured profile assembly. The fixed set of 24 questions across eight personality dimensions provides an organizational framework, but the paper does not clearly demonstrate a substantial technical innovation beyond integrating these existing components.
Questions for authors
In addition to the weakness,
- What are the LLM sources? Are they the 4 open-source backbones? or the gpy-5.6-sol ?
- How are Big Five scores generated? is table 3 averaging over samples and judges?
5 comments
Response to Reviewer AwyG - Part II
3. Response to Weakness 3: Technical Contribution vs. Minimal Step Implementations
We do not claim to have invented a new underlying retrieval algorithm or summarization model. As introduced in Lines 105-117, our core contributions lie in defining a new practical problem class (role-playing subjects without gold-standard profiles or pretrained identity memory) and proposing a general, psychology-grounded profile reconstruction design paradigm along with a four-level controlled evaluation protocol.
When validating this new paradigm in our experiments, we deliberately adopted the simplest and most direct implementation for every step of the paradigm. This design choice serves two key scientific purposes:
- Eliminating confounding variables: It demonstrates that the significant downstream utility gains (Table 2), the 100% dimension coverage rate (Table 4), and the high character distinctiveness (Table 3) genuinely stem from the 8-dimension, 24-question identity-free reconstruction paradigm itself, rather than from specialized engineering tricks.
- Providing a clean, transparent baseline for future research: As diagnosed in Section 4.4, third-party comments exhibit a 4.7% to 10.2% contradiction rate against official profiles. Our minimal paradigm implementation establishes a clean baseline pipeline upon which future work can specifically design consensus-weighting and conflict-resolution algorithms.
4. Responses to Questions for Authors
- Question 1 (What are the LLM sources?): As detailed in Item 4 of our response to Weakness 2 above,
Source = LLMusesgpt-5.6-solqueried by character name to answer the 24 profiling questions, whereas the 4 open-source backbones are used exclusively as downstream role-playing Actors. - Question 2 (How are Big Five scores generated, and what does Table 3 report?):
Big Five scores are generated by the independent evaluator Judge B (
Kimi-K2.6), which did not participate in building any profiles or generating any actor responses. For each of the 16 subjects and each type of rated material (Official profile, Minstrel profile, and aggregated Actor responses), Judge B rates the subject on a 9-point scale (1 to 9) across each of the five Big Five factors (3 independent repeated ratings per cell, yielding rating observations per factor with a within-cell standard deviation of 0.63). Table 3 does not average across multiple judges. Instead, for each factor , Table 3 reports the Intraclass Correlation (, Equation 7) from a one-way ANOVA across subjects, which measures the proportion of total rating variance that lies between different subjects relative to repeated-rating noise.
Dear Reviewer AwyG,
We wanted to briefly follow up to let you know that all revisions promised in our response have now been incorporated into the updated PDF (marked in red):
- Streamlined Section 3 & concrete running example: We removed the composite stage equation, kept only lightweight stage-level signatures (), moved concrete operator settings to Section 4.1, updated Figure 2, and added the new Table 2 (Section 3.5) tracing one profiling question through all four stages with real data.
- Complete experimental & judging protocols: Section 4.1 and Appendix C (Items and filtering, plus verbatim system prompts) now report the RoleAgentBench item structure, the sequential filtering counts ( items), the exact actor instructions and JSON format for the no-information control (), and the verbatim pairwise judge prompt and rubric. Appendix D (Table 13) explicitly separates the
Profiler(gpt-5.6-sol), the four open-sourceActors, and the two cross-familyJudges. - Clarified positioning: Sections 1 and 3 clarify that each stage uses its most direct instantiation on purpose so that the empirical gains isolate the problem formulation and psychological interface rather than operator engineering.
We hope these updates resolve your concerns regarding readability and reproducibility, and we remain happy to address any further questions.
Response to Reviewer AwyG - Part I
We thank Reviewer AwyG for recognizing the practical value of our problem formulation, our four-dimensional psychological evaluation framework, and our empirical findings.
1. Response to Weakness 1: Formal Notation in Section 3 and Adding a Concrete Running Example
Because this paper introduces a general design paradigm rather than a dataset-specific script, the lightweight mathematical notation in Section 3 (the information need set N, dimension map d, and stage operators A,C,G) serves an important structural purpose. It defines the formal interface signatures that allow different information sources (Principle P2) and conversion strategies (Principle P3) to be swapped without altering downstream stages, and it connects Section 3 directly to the evaluation metrics in Section 4 (W(c),ρf,cov(c),ε(c)) and the constrained objective in Appendix B (Equation 10, Line 733). These signatures give future researchers a precise specification to follow when extending the paradigm.
At the same time, we fully agree with your suggestion that introducing these concepts alongside a concrete example makes Section 3 much easier to follow. In the revised manuscript, we will streamline the prose in Section 3 and add a concrete running example across Sections 3.2 to 3.5 (tracing how Question D5-Q2 for
hp_mcgonagallretrieves raw comments mn, converts them into de-identified personality cues or summaries un, and assembles the final profile section cs).2. Response to Weakness 2: Implementation Details and Experimental Setup
Below are the exact implementation and evaluation protocols (which we will add to Appendix C):
knowledge_requirement != essential: excluded 118 items that required recalling unseen canonical plot events or franchise-specific terms absent from the visible prompt.input_sufficiency == sufficient: excluded 118 items whose visible input alone could not determine the response direction.profile_relevance != none: ensured that the item tests stable personality traits, retaining 95 items across 10 characters.B-none), the Actor receives the same message as in the informed conditions, but the conditioning field is an empty string. The fixed system instruction is: "Role-play the target described below. The target's identity is deliberately withheld: do not guess it, do not state it, and do not name any specific real or fictional work. Do not mechanically list the supplied traits. Do not invent biographical facts that the supplied material does not support. Respond naturally to the situation in the target's expressive style and behavioural pattern." The user message is a JSON object with the fieldsidentity(the wordwithheld),conditioning,contextandquery. Decoding uses model defaults with sampling, up to 512 new tokens.CANDIDATE AandCANDIDATE B, with condition labels hidden. Every pair is judged independently in both display orders (ABandBA), and any pair whose verdict changes with order is recorded asundecided.tie.LLMsources (Source = LLM)?Source = LLM(theLLMrows in Table 2) uses the external modelgpt-5.6-sol(see Line 291 and theProfilerrow in Appendix D Table 10), queried by the character's real name with the 24 profiling questions (prompt in Appendix C, Lines 814-821) to test eliciting a profile from an external LLM with prior knowledge of the character (such as a user's companion agent). By contrast, the 4 open-source models (Qwen2.5-7B/14B,Llama-3.1-8B,Mistral-7B) serve strictly as downstream Actors and never build profiles or judge responses.