acceptodds
Withdrawn from ICLR 2027

FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As large language models (LLMs) advance in role-playing (RP) tasks, existing benchmarks quickly become obsolete due to their narrow scope, outdated interaction paradigms, and limited adaptability across diverse application scenarios. To address the benchmark limitations, we introduce FURINA-Builder, a novel multi-agent collaboration pipeline that automatically constructs fully customizable RP benchmarks at any scale. As the first benchmark builder in the RP domain designed for adaptable assessment, FURINA-Builder enables evaluation of arbitrary characters across diverse scenarios and prompt formats. From our builder, we derive FURINA-Bench, a comprehensive new role-playing benchmark generated by 20+ strong source LLMs, featuring both established and synthesized test characters, each assessed with dimension-specific evaluation criteria. Human evaluation and preliminary separability analysis justify our pipeline and benchmark design. Based on FRUINA-Bench, we discover key trade-offs between reasoning, hallucination, and RP performance across cutting-edge LLMs (general and dedicated RP models) and character types. Together, these findings demonstrate the effectiveness of FURINA-Builder and the challenge posed by FURINA-Bench, establishing a strong foundation for future research on RP evaluation.

Primary area:
datasets and benchmarks
Keywords:
Role-playing (RP), Game NPC, Automatic Benchmark Construction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QRgKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents FURINA-Builder, an LLM-orchestrated pipeline for automatically constructing customizable role-playing benchmarks from user-defined character profiles and book-derived multi-party dialogue scenarios. A director model selects the next speaker, scene models generate the surrounding conversation, source models and a fixed GPT-4.1 baseline produce candidate responses, and a GPT-4.1 judge assigns one of five evaluation dimensions and selects the stronger response during benchmark construction.

Using this pipeline, the authors release FURINA-Bench, a bilingual benchmark containing established and synthesized characters, multi-character scenarios, and dimension-specific test turns covering context reliance, factual recall, reflective reasoning, conversational ability, and preference alignment. The paper evaluates a range of general-purpose, reasoning, and role-playing LLMs through bidirectional pairwise comparisons against the fixed baseline. It reports findings about model scale, reasoning, character type, hallucination, and the trade-off between RP performance and reliability. Human annotation studies, judge replacement experiments, and a separability comparison are provided as validation.

Critical strengths
  1. The paper addresses a timely and practically relevant problem: role-playing benchmarks quickly become stale when they are fixed to a small number of characters, dialogue formats, and evaluation dimensions. FURINA-Builder provides a reasonably clear framework for specifying characters, visibility, scenarios, and dimension-specific evaluation criteria. The resulting bilingual, multi-party benchmark with both established and synthesized characters could be useful to researchers studying role-playing agents.

  2. The paper provides a relatively broad empirical study over general-purpose, reasoning, and dedicated role-playing models. It also attempts several forms of validation, including human annotation for dimension selection and pairwise judgment, judge-model replacement, and a separability comparison. The illustrative examples and high-level system diagrams make the intended benchmark workflow easy to understand.

Critical weaknesses
  1. The benchmark construction and evaluation are circular and source-conditioned. During construction, GPT-4.1 serves as both the base model and the judge, and the pipeline explicitly retains only turns where a source model is preferred to GPT-4.1. The released JSON files contain only construction scores 1 and 2, with no retained source-loss or tie cases. The same GPT-4.1 response is then used as the fixed baseline for evaluating all models. Furthermore, the evaluation prompt gives the test model a dimension-specific response strategy, whereas the stored baseline is reused without a matched regeneration under the same strategy. Consequently, the benchmark score is not an unbiased measure of general RP ability; it primarily measures performance relative to one GPT-4.1 anchor under a source-conditioned and prompt-assisted protocol. Judge replacement with only four representative models does not remove this construction bias. The paper needs independent bases/judges, prompt-matched comparisons, source-balanced or unfiltered data, and an analysis of raw construction outcomes.

  2. The validation, statistics, and released implementation do not support the paper’s strongest claims. The hallucination metric is derived from keyword detection in judge CoT outputs rather than direct checking of the model response against the dialogue context. The main text mentions a Qwen2.5-32B-Instruct checker, while Appendix O describes a DeepSeek-R1 meta-verification pipeline. Human validation reports Pearson correlations and raw agreement rates but does not provide annotator-level agreement, confusion matrices, precision/recall, confidence intervals, or enough sampling details. The reported 95% confidence intervals have unexplained units and computation procedures. The supplementary package also contains reproducibility failures: the evaluation shell scripts call a missing nalobench_evaluation_test_only.py, the Builder scripts reference missing model configuration files, the advertised synthesis pipeline is absent, and the released JSON counts and dimension labels do not match the paper. The Chinese data contains one nonstandard CONVERSATIONAL label that the released evaluator does not support and silently treats as CR. These issues make the reported numbers and the claims about scalability, hallucination, and reliability difficult to verify.

Questions for authors
  1. Can the authors provide an evaluation with a prompt-matched baseline and test-model condition, together with an independent base/judge model, and without filtering the benchmark to only source-model wins over GPT-4.1? How much do the rankings, reasoning effects, established-versus-synthesized gaps, and hallucination trends change under this protocol?

  2. What exactly is the estimator behind the reported 95% confidence intervals? Please specify the score scale, resampling method, clustering unit, number of samples, and whether model/API/judge randomness is included. Please also reconcile the reported totals with the released data: 7,181 versus 7,180 evaluated turns, 1,459 versus 1,494 conversations, and the nonstandard CONVERSATIONAL label in the Chinese JSON.

  3. Which model and exact prompt are used for hallucination checking? Why does the main text describe a Qwen2.5-32B-Instruct checker while Appendix O describes DeepSeek-R1? Can the authors provide context-level human labels, confusion matrices, precision/recall, annotator agreement, and the complete checking code?

  4. What permissions, licenses, or provenance documentation support releasing the book-derived scene pool and dialogue material based on 100 English and 80 Chinese books? The paper states that no raw novel content remains, but the appendix and released examples contain book-derived dialogue that appears close to source passages. Please clarify the transformation and filtering procedure and the intended release/removal policy.

Reviewer_L5pHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses a "what if" style synthetic scenario evaluation for Role Play capability of tested LLMs. By providing necessary underlying motivation, character list and setup, a tested LLM is asked to perform in situ and its behaviour will be evaluated in that context. The tests are performed across the board on both open-weight and closed-weight models and multi-dimensionality evaluations are done.

Critical strengths

The paper has done extensive evaluations of multiple models, across a series of models. The line of development is natural and follows clear motivation.

Critical weaknesses

This paper seems to belong to the persona-following line of work, instead of RP foundational capability work. Persona-following puts more emphasis on instruction following, and the metrics tested in the paper are mostly instruction-following oriented. E.g., style consistency is not heavily evaluated in the paper, so potential reward hacking behaviors of the model include exploiting surface-level context clues or repeating clichés and catchphrases. The models under evaluation are rather outdated, all pre-GPT-4.1 models (though it is true that a lot of research were done then). The paper would benefit from mining more reward-hacking patterns and developing methods to suppress them. Besides, the evaluation setup resembles harness, yet it is not revamped under modern harness logic. For example, there is no CLI interaction for the model to access external knowledge; some tasks end up as unnecessary memory tests that strongly favor large models.

Questions for authors

How to prevent the sole factor of model world-knowledge possession from dominating the tests?

Please comment on these works: https://arxiv.org/abs/2502.16940 https://arxiv.org/abs/2508.10014

Why in Figure 6, reliability of gpt-4o exceeds 100?

Reviewer_paJYOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FURINA-Builder, a pipeline for constructing role-playing benchmarks from configurable character profiles and literary scenes. Multiple models simulate conversations, a judge selects an evaluation dimension for each test turn, and candidate responses are compared with a reference model. The resulting FURINA-Bench contains 7,181 test utterances covering 20 established and synthesized characters in English and Chinese. Models are evaluated through dimension-specific pairwise comparisons against GPT-4.1. The experiments examine model rankings, the effects of role-playing training and reasoning, and differences between established and synthesized characters. Human studies assess dimension selection and pairwise judgments. The authors also analyze hallucination and its relationship with role-playing performance.

Critical strengths

The most useful contribution is the ability to construct evaluations for user-specified characters within multi-party scenes. The distinction between public and private profile attributes is also sensible for this setting. Bringing established and synthesized characters into the same construction framework makes the resource useful beyond a fixed list of familiar fictional roles. The bilingual benchmark and the range of source models give the proposal a substantial empirical basis. The authors provide several useful checks on the automatic evaluation. Human annotation covers 1,000 dimension-selection examples and 400 pairwise-scoring examples, with average selection accuracy of 0.892 and average judge–human Pearson correlation of 0.6275 (Section 4.3; Tables 10–11). Appendix N preserves the ordering of four models across three judges in both languages. Figure 4 also compares Character-LLM and Neeko training on the same Qwen3-8B backbone. Together, these checks provide substantive support for the benchmark’s utility and its ability to detect changes from role-playing training.

Critical weaknesses

The hallucination results need a clearer connection to their validation. Section 5.5 and Appendix V count a case only when the judge explanations for both response orders identify hallucination; Table 25 asks the checker to extract this finding from the explanations. Appendix O instead describes DeepSeek-R1 generating and verifying a hallucination analysis. I could not determine whether the human study validates the procedure used to produce Table 16 and Figures 5–6. The agreement rates of 98.5% and 97.0% are encouraging, but agreement alone does not establish sensitivity to infrequent errors. Positive-case counts and detection precision and recall would make this evidence interpretable. The authors already acknowledge the Claude exception in Appendix W: thinking improves its Chinese role-playing score while lowering both hallucination rates. The broader claim that reasoning increases hallucination should be qualified accordingly. The evidence for improved model separation is less conclusive than the paper suggests. Figure 2 and the separation index in Appendix R describe the distribution of scores across four models; they do not establish that the distinctions are more reliable than those made by GCA. Equation (2) places extra weight on strong wins, so a comparison with conventional win rates would help establish how much the conclusions depend on this choice. Appendix N is reassuring for the four tested models, but does not settle the ordering of close comparisons such as DeepSeek-R1 and Claude Sonnet 4 in English. Table 1 reports confidence intervals, yet their units and the treatment of turns from a shared dialogue are unclear. These concerns limit the interpretation of small differences rather than invalidate the broad performance trends.

Questions for authors

Which hallucination procedure produced Table 16 and Figures 5–6, and was that exact procedure evaluated in Appendix O? Please clarify what the annotators inspected and report the number of positive cases and precision and recall against direct human assessment of the responses. Please also confirm the denominators used for the EC and SC rates, since these evaluate different character and dimension subsets. For the confidence intervals in Table 1, what is the resampling unit, and are the bracketed values deviations on the normalized 0–1 scale while the scores are percentages? If individual turns were resampled, a check that groups turns from the same dialogue would be useful. For the closest model comparisons, would changing the judge or using conventional win rates with half credit for ties alter the ordering? These checks would clarify which small differences support a substantive comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.