- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper presents FURINA-Builder, an LLM-orchestrated pipeline for automatically constructing customizable role-playing benchmarks from user-defined character profiles and book-derived multi-party dialogue scenarios. A director model selects the next speaker, scene models generate the surrounding conversation, source models and a fixed GPT-4.1 baseline produce candidate responses, and a GPT-4.1 judge assigns one of five evaluation dimensions and selects the stronger response during benchmark construction.
Using this pipeline, the authors release FURINA-Bench, a bilingual benchmark containing established and synthesized characters, multi-character scenarios, and dimension-specific test turns covering context reliance, factual recall, reflective reasoning, conversational ability, and preference alignment. The paper evaluates a range of general-purpose, reasoning, and role-playing LLMs through bidirectional pairwise comparisons against the fixed baseline. It reports findings about model scale, reasoning, character type, hallucination, and the trade-off between RP performance and reliability. Human annotation studies, judge replacement experiments, and a separability comparison are provided as validation.
Critical strengths
-
The paper addresses a timely and practically relevant problem: role-playing benchmarks quickly become stale when they are fixed to a small number of characters, dialogue formats, and evaluation dimensions. FURINA-Builder provides a reasonably clear framework for specifying characters, visibility, scenarios, and dimension-specific evaluation criteria. The resulting bilingual, multi-party benchmark with both established and synthesized characters could be useful to researchers studying role-playing agents.
-
The paper provides a relatively broad empirical study over general-purpose, reasoning, and dedicated role-playing models. It also attempts several forms of validation, including human annotation for dimension selection and pairwise judgment, judge-model replacement, and a separability comparison. The illustrative examples and high-level system diagrams make the intended benchmark workflow easy to understand.
Critical weaknesses
-
The benchmark construction and evaluation are circular and source-conditioned. During construction, GPT-4.1 serves as both the base model and the judge, and the pipeline explicitly retains only turns where a source model is preferred to GPT-4.1. The released JSON files contain only construction scores 1 and 2, with no retained source-loss or tie cases. The same GPT-4.1 response is then used as the fixed baseline for evaluating all models. Furthermore, the evaluation prompt gives the test model a dimension-specific response strategy, whereas the stored baseline is reused without a matched regeneration under the same strategy. Consequently, the benchmark score is not an unbiased measure of general RP ability; it primarily measures performance relative to one GPT-4.1 anchor under a source-conditioned and prompt-assisted protocol. Judge replacement with only four representative models does not remove this construction bias. The paper needs independent bases/judges, prompt-matched comparisons, source-balanced or unfiltered data, and an analysis of raw construction outcomes.
-
The validation, statistics, and released implementation do not support the paper’s strongest claims. The hallucination metric is derived from keyword detection in judge CoT outputs rather than direct checking of the model response against the dialogue context. The main text mentions a Qwen2.5-32B-Instruct checker, while Appendix O describes a DeepSeek-R1 meta-verification pipeline. Human validation reports Pearson correlations and raw agreement rates but does not provide annotator-level agreement, confusion matrices, precision/recall, confidence intervals, or enough sampling details. The reported 95% confidence intervals have unexplained units and computation procedures. The supplementary package also contains reproducibility failures: the evaluation shell scripts call a missing
nalobench_evaluation_test_only.py, the Builder scripts reference missing model configuration files, the advertised synthesis pipeline is absent, and the released JSON counts and dimension labels do not match the paper. The Chinese data contains one nonstandardCONVERSATIONALlabel that the released evaluator does not support and silently treats as CR. These issues make the reported numbers and the claims about scalability, hallucination, and reliability difficult to verify.
Questions for authors
-
Can the authors provide an evaluation with a prompt-matched baseline and test-model condition, together with an independent base/judge model, and without filtering the benchmark to only source-model wins over GPT-4.1? How much do the rankings, reasoning effects, established-versus-synthesized gaps, and hallucination trends change under this protocol?
-
What exactly is the estimator behind the reported 95% confidence intervals? Please specify the score scale, resampling method, clustering unit, number of samples, and whether model/API/judge randomness is included. Please also reconcile the reported totals with the released data: 7,181 versus 7,180 evaluated turns, 1,459 versus 1,494 conversations, and the nonstandard
CONVERSATIONALlabel in the Chinese JSON. -
Which model and exact prompt are used for hallucination checking? Why does the main text describe a Qwen2.5-32B-Instruct checker while Appendix O describes DeepSeek-R1? Can the authors provide context-level human labels, confusion matrices, precision/recall, annotator agreement, and the complete checking code?
-
What permissions, licenses, or provenance documentation support releasing the book-derived scene pool and dialogue material based on 100 English and 80 Chinese books? The paper states that no raw novel content remains, but the appendix and released examples contain book-derived dialogue that appears close to source passages. Please clarify the transformation and filtering procedure and the intended release/removal policy.