- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission constructs a benchmark, focusing on NPC role play. It examines whether language models can satisfy both state sensitivity and character continuity: acting as the current state requires while preserving character identity and behavioral boundaries. The proposed NPCBank benchmark consistss of 7 games, 37 NPCs, covering more than 20000 cases and eximaning 21 capabilities.
To evaluate role-playing performance, the authors define two requirements: obligations specify what a response should accomplish, while invariants specify the constraints it must preserve. For each instance, the authors specify the current character state and provide separate model and judge views, evaluating the generated response against the state-conditioned obligations and invariants. The experiments and analysis address three challenges, i.e., supplied-state role-playing, consumer-GPU inference, and role-play persistence over multiple turns of interactions.
Critical strengths
- The benchmark is solidly constructed, and provides valuable resource to the field.
- The experiments and analyses are thorough, which cover a broader range of fine-grained evaluation.
Critical weaknesses
- The rationale for the capability taxonomy and probe remain unclear. Appendix B provides operational definitions, scope boundaries, and relations between capabilities. However, the paper does not clearly explain how the seven domains and 21 capabilities were selected, or what guided their inclusion and level of granularity. Similarly, Section D.4 reports 85 distinct probe keys without providing a complete inventory of their definitions and mappings to capabilities. This makes it difficult to assess the behavioural coverage beyond the number of cases, particularly whether different probe keys represent substantively different tests.
Questions for authors
Please see above.
2 comments
Thank you for the response. I think the current score is fair, and I will keep it.
Response to Reviewer T539
Thank you for the careful review and for recognizing the benchmark and our experiments. We really appreciate it. Below we explain how the taxonomy and the probes were built, and what we will add to the paper.
W1a: How were the seven domains and 21 capabilities chosen?
Thank you for pointing this out. We agree that the paper should explain this. By design, the taxonomy starts from what goes wrong when an NPC talks to a player: a character slipping out of its voice, citing facts from the wrong point in the story, revealing what it should keep back, ignoring who it is talking to, forgetting what was said earlier, or dropping a task it had agreed to. We catalogued 34 such failure families (Table 9) and defined 36 capabilities to cover them, grouped into seven domains by what a failure is about: the character's persona, the game world, what it knows and may reveal, its relationships, the conversation, its goals and commitments, and pressure from the player.
One rule then set both inclusion and granularity: we kept a capability only if a judge could check it on its own, with its own required behavior, failure mode and scope, and merged capabilities that could not be told apart. That left 21, of which 8 are judged on observable behavior and 13 against source evidence. Merged capabilities were not dropped. They stay in the registry as aliases and, where they define a distinct test, as probes. For example, identity coherence has probes for identity, motivation and values.
We also checked coverage against existing role-play benchmarks. The main dimensions of CharacterEval, CharacterBench and SocialBench (memory, knowledge, persona, emotion, consistency, social awareness) all fall within these domains. Beyond them, our taxonomy makes explicit the behaviors that depend on the current game state, such as what may be disclosed to whom, where the story currently is, and which commitments are still open.
In the revision we will:
W1b: A complete inventory of the 85 probe keys, and whether they are different tests.
We built the complete probe inventory you asked for. Of the 85 probe keys listed in D.4, 8 are shortened forms of longer keys, so there are 77 distinct probes. Probes that check the same behavior under the same conditions share a test design, and the 77 probes use 64 designs. Different probes also behave differently in practice. Of the 18 capabilities with more than one probe, 14 have enough cases to compare, and in 8 of these the probe significantly changes difficulty (permutation test). For example, mean scores of the manipulation-resistance probes range from 0.14 to 0.55. In the other 6 we find no difference in difficulty. Five capabilities currently have a single test design: initiative regulation, calibration, timeline grounding, commitment tracking and factual grounding.
In the revision we will:
Summary. The taxonomy comes from recurring NPC failure modes, and one rule, that a judge can check each capability on its own, sets which capabilities are kept and at what granularity. The 77 probes use 64 test designs and change difficulty in practice. The revision will add the derivation, the benchmark comparison, a complete probe table, and probes for the five single-design capabilities. We hope this addresses your concerns, and we are happy to provide further details.