acceptodds
Under review as a conference paper at ICLR 2027

The Non-Player Character Bank

Abstract

A designed world, however beautiful, is still a script. Open-world games have mastered immersion through narrative, art, and music, yet their inhabitants remain bounded by authored conversations and relationship branches. Game players want more: randomness, free conversation and dynamic relationships. Large language models have begun to loosen the script, but between a model that talks and a character that lives, the distance is still wide. We present the Non-Player Character Bank (NPCBank), an ecosystem for crossing it. NPCBank Data grounds characters in source evidence and turns them into structured dialogue cases; NPCBank Web makes the resulting characters, evidence, and evaluation contracts open to inspection and annotation; and NPCBank Lab presents a demo game for testing interactions with model-based NPCs. NPCBank is based on a capability ontology, a contract-based case schema, and a construction agent that turns source material into character representations and dialogue cases. The first release contains 20,655 cases for 37 NPCs across seven games, covering 21 dialogue capabilities. Building on NPCBank, we formulate three challenges: supplied-state role-play, consumer-GPU inference, and long-horizon persistence. We find that models fail on different capabilities, consumer configurations expose distinct quality–memory and quality–latency trade-offs, and role-play remains fragile over long interactions. We release NPCBank as a foundation for building interactive characters across computational character modeling and game AI.

Primary area:
datasets and benchmarks
Keywords:
Role-playing agents, NPC dialogue, LLM-as-a-judge, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T539OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission constructs a benchmark, focusing on NPC role play. It examines whether language models can satisfy both state sensitivity and character continuity: acting as the current state requires while preserving character identity and behavioral boundaries. The proposed NPCBank benchmark consistss of 7 games, 37 NPCs, covering more than 20000 cases and eximaning 21 capabilities.

To evaluate role-playing performance, the authors define two requirements: obligations specify what a response should accomplish, while invariants specify the constraints it must preserve. For each instance, the authors specify the current character state and provide separate model and judge views, evaluating the generated response against the state-conditioned obligations and invariants. The experiments and analysis address three challenges, i.e., supplied-state role-playing, consumer-GPU inference, and role-play persistence over multiple turns of interactions.

Critical strengths
  1. The benchmark is solidly constructed, and provides valuable resource to the field.
  2. The experiments and analyses are thorough, which cover a broader range of fine-grained evaluation.
Critical weaknesses
  1. The rationale for the capability taxonomy and probe remain unclear. Appendix B provides operational definitions, scope boundaries, and relations between capabilities. However, the paper does not clearly explain how the seven domains and 21 capabilities were selected, or what guided their inclusion and level of granularity. Similarly, Section D.4 reports 85 distinct probe keys without providing a complete inventory of their definitions and mappings to capabilities. This makes it difficult to assess the behavioural coverage beyond the number of cases, particularly whether different probe keys represent substantively different tests.
Questions for authors

Please see above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer T539

Thank you for the careful review and for recognizing the benchmark and our experiments. We really appreciate it. Below we explain how the taxonomy and the probes were built, and what we will add to the paper.

W1a: How were the seven domains and 21 capabilities chosen?

Thank you for pointing this out. We agree that the paper should explain this. By design, the taxonomy starts from what goes wrong when an NPC talks to a player: a character slipping out of its voice, citing facts from the wrong point in the story, revealing what it should keep back, ignoring who it is talking to, forgetting what was said earlier, or dropping a task it had agreed to. We catalogued 34 such failure families (Table 9) and defined 36 capabilities to cover them, grouped into seven domains by what a failure is about: the character's persona, the game world, what it knows and may reveal, its relationships, the conversation, its goals and commitments, and pressure from the player.

One rule then set both inclusion and granularity: we kept a capability only if a judge could check it on its own, with its own required behavior, failure mode and scope, and merged capabilities that could not be told apart. That left 21, of which 8 are judged on observable behavior and 13 against source evidence. Merged capabilities were not dropped. They stay in the registry as aliases and, where they define a distinct test, as probes. For example, identity coherence has probes for identity, motivation and values.

We also checked coverage against existing role-play benchmarks. The main dimensions of CharacterEval, CharacterBench and SocialBench (memory, knowledge, persona, emotion, consistency, social awareness) all fall within these domains. Beyond them, our taxonomy makes explicit the behaviors that depend on the current game state, such as what may be disclosed to whom, where the story currently is, and which commitments are still open.

In the revision we will:

  • Add this derivation to Appendix B, including which capabilities were merged into each of the 21.
  • Add the comparison with existing role-play benchmarks.

W1b: A complete inventory of the 85 probe keys, and whether they are different tests.

We built the complete probe inventory you asked for. Of the 85 probe keys listed in D.4, 8 are shortened forms of longer keys, so there are 77 distinct probes. Probes that check the same behavior under the same conditions share a test design, and the 77 probes use 64 designs. Different probes also behave differently in practice. Of the 18 capabilities with more than one probe, 14 have enough cases to compare, and in 8 of these the probe significantly changes difficulty (permutation test). For example, mean scores of the manipulation-resistance probes range from 0.14 to 0.55. In the other 6 we find no difference in difficulty. Five capabilities currently have a single test design: initiative regulation, calibration, timeline grounding, commitment tracking and factual grounding.

In the revision we will:

  • Add a complete probe table to D.4, giving each probe's capability, what it asks the model to do, which design it shares, and how many cases and NPCs it covers.
  • Correct the count to 77 distinct probes. No case changes.
  • Report the probe comparison for every capability where it can be run, including those where probes do not differ.
  • Name the five single-design capabilities, and add probes for them in the next version of NPCBank.

Summary. The taxonomy comes from recurring NPC failure modes, and one rule, that a judge can check each capability on its own, sets which capabilities are kept and at what granularity. The 77 probes use 64 test designs and change difficulty in practice. The revision will add the derivation, the benchmark comparison, a complete probe table, and probes for the five single-design capabilities. We hope this addresses your concerns, and we are happy to provide further details.

Thank you for the response. I think the current score is fair, and I will keep it.

Reviewer_taz8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces NPCBank, a benchmark for evaluating game NPC role-playing through state-dependent obligations and invariant constraints. It contains 20,655 cases covering 37 NPCs, seven games, and 21 capabilities. Three challenges assess supplied-state role-playing, consumer-GPU inference, and persistence over 24-turn interactions. Supported by community case validation and judge audits, the experiments reveal capability-specific weaknesses, quality–resource trade-offs, and differences between maintaining disclosure boundaries and fulfilling conversational obligations.

Critical strengths
  1. Interpretable evaluation design.
    Separating obligations from invariants distinguishes task fulfillment from preservation of character identity and behavioral boundaries. The matched-state experiments provide a meaningful intervention: removing the target-state information reduces reported adaptation by 22.2–31.4 percentage points (Table 60), supporting the evaluation’s sensitivity to supplied state.

  2. Substantial resource and validation effort.
    The benchmark provides detailed provenance, construction safeguards, and reproducibility documentation. The community study finds 92.4% of 370 sampled cases valid. Human-reference audits, cross-family judging, paired comparisons, and NPC-level uncertainty estimates further strengthen the empirical foundation.

Critical weaknesses
  1. Evidence for character-specific and evolving-state competence remains limited.
    Most cases are static-conditioned (18,737/20,655), and the persistence protocol fixes persona, knowledge, relationship, and timeline. These settings provide controlled tests, but leave unclear how much performance depends on character-specific behavior beyond following supplied facts and instructions. Although consistent with the stated supplied-state scope, this limits support for broader claims about evolving NPCs. A persona-ablation baseline and separate results for dynamic subsets would strengthen construct validity.
Questions for authors
  1. How do results change when persona/voice information is removed while facts and current state remain? How do the pre/post and trajectory subsets compare with static cases?
  2. Can you provide blinded human validation of pairs that pass individually but fail adaptation, distinguishing genuine behavioral errors from additional rubric requirements or judge inconsistency?
2 comments
AuthorsOpenReview ↗

Response to Reviewer taz8

Thank you for the careful review and the concrete suggestions. We ran each analysis you proposed, and below we report the results and what we will change.

W1 and Q1a: How much does performance depend on the character itself?

This is an excellent question and one we also asked ourselves when we began the experiments! By design, NPCBank asks whether a model can play a character well under the conditions a game supplies, so drawing on the supplied information is the task rather than a shortcut. What we can test is whether the persona part of that information matters beyond the facts. In our inputs, persona and voice are described in exactly two fields (style anchors and voice behaviors). We removed them, kept everything else (name, facts, state and dialogue), and re-ran 168 cases (8 per capability) on 4 open models with the same seeds. Scores are Q under the Gemma judge, with 95% CIs on the drop:

With personaWithoutDrop (points)
All cases0.8310.7716.1 [3.4, 9.0]
Social capabilities0.8240.64717.7 [9.0, 27.7]
Persona capabilities0.9090.8911.8 [−6.2, 11.2]

Removing the persona lowers scores by about 6 points overall and by about 18 points on social capabilities, such as how the character treats the other person. So performance depends on character information beyond the supplied facts. Persona-capability scores barely move, probably because the models also recognize these characters by name.

In the revision we will:

  • Add this ablation to the paper.
  • Add a variant that also hides the character's name, to separate recognition by name from the supplied persona.
  • Add more criteria that check persona directly.

W1 and Q1b: Dynamic cases and how they compare with static ones.

Static cases keep the supplied state fixed. Pre/post cases change it between two points, and trajectory cases let it evolve over several steps. Most cases are static by design, but they still test the state, since what a reply must do depends on the declared state. NPCBank has since been updated to version 1.0.7 (26,704 cases, 80 NPCs, eight games). It has 2,457 dynamic cases, up from 1,918, plus 290 contrast groups, 124 of which change one part of the state at a time (Q2). On the submitted version, our panel has 76 dynamic cases:

SubsetQ (Gemma)Q (Qwen)Cases
Static0.730.72700
Pre/post0.690.6855
Trajectory0.760.7221

Under both judges, pre/post cases score about 4 points below static ones, and trajectory cases about the same. We treat this as a first look.

In the revision we will:

  • Raise the share of pre/post and trajectory cases in the next version and in the evaluation panel, and report the three subsets separately.
  • State the scope as supplied-state role-play. In the abstract and where the persistence protocol is introduced, say that it fixes the state on purpose, to measure how long a model stays in character.
  • Add more contrasts on the conversation history, which our pair study found hardest for models (see Q2).

Q2: Pairs that pass individually but fail adaptation.

As in our reply to Reviewer mVuY, we re-ran the paired study with swap and removal baselines on 290 contrast groups from version 1.0.7, and the removal effect you highlight still holds, at 18–21 points. In this set, 334 pairs have both replies passing their own contracts. In a separate audit, blind to model and judge, one author checked 12 of these that a judge flagged, all on history contrasts, mixed with 12 random history pairs that both judges passed. Of the 12 flagged pairs, 10 were genuine behavioral errors rather than extra rubric requirements or judge inconsistency, and 2 were false alarms. The error is the same each time: in the version of the conversation where something was never said, the reply still "remembers" it, which that version's contract rules out. The author found the same error in 10 of the 12 unflagged pairs. So on history contrasts the flags are mostly correct but incomplete, and the failure rates we report are a lower bound.

In the revision we will:

  • Estimate the rate of these failures from a larger human check, with a second annotator.
  • Add an explicit judge check for unsupported recall, that is, referring to something that was never said, and re-score history cases with it.
  • Release the audit labels together with the new contrast set.

Summary. Within the supplied-state setting, persona information matters beyond the facts, most clearly in social behavior. Pairs that pass individually but fail adaptation are genuine errors, common on history contrasts, where our judges under-count them. The revision will add the persona ablation, a name-hidden variant, a larger share of dynamic cases, more history contrasts, and an explicit recall check. We hope this addresses your concerns, and we are happy to provide further details!

Thank you for your reply. It has resolved my main concerns. I will keep this positive score.

Reviewer_mVuYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents NPCBank, a benchmark for role-play in which the character’s current state (relationship, story phase, scene, what the character knows and may reveal) is supplied explicitly, and each case is scored against a contract derived from that state. The contract lists obligations the reply should meet and invariants it must not break. Cases are compiled by an LLM pipeline from game sources, giving 20,655 cases for 37 NPCs across seven games under an ontology of 21 capabilities. The evaluation covers state-conditioned behavior on single cases and on minimal pairs that change one state axis. Scoring relies on an LLM judge calibrated against adjudicated human labels.

Critical strengths
  1. The case contract is the paper’s key contribution. Splitting what a reply has to do from what it must not violate, and tying both to an explicit state, catches failures that benchmarks focused on profile fidelity can’t see. For example, a reply can sound exactly like the character and still tell a guarded stranger something she’d only tell a partner. The matched pairs extend this idea across states. For the four models tested, 11-14% of groups in which both replies pass their individual contracts still miss the change (or preservation) the contracts require. Single-case scoring cannot see these failures, so the paired design is worth having, although these labels come from the pair-level judge I question in Weakness 2.

  2. The measurement work is careful. The judge was selected with a predeclared gate, audited against independently labeled human references and a second judge family, and the paper openly reports where it disagrees with humans. Most cases carry source evidence in the judge view, and the released traces and labels make the issues below checkable.

Critical weaknesses
  1. The absolute JPR numbers still look unreliable. Table 20 shows clear inflation: Gemma gives 67.5% strict pass versus 53.8% for the adjudicated reference. More importantly, the Gemma–Qwen gap varies a lot across systems (2.95 to 15.95 points in Table 19), so judge bias does not seem uniform. This makes absolute claims like “91.6%” or “near the ceiling,” as well as mid-ranking differences, hard to trust. The current audit is too small to estimate this bias per system.

  2. The “models read the state but under-react to it” claim needs stronger validation. The pairwise labels come from Gemma-4-31B, but I could not find a human audit of this judge. The removal control is also noisy: after removing the target axis, prompts are identical, yet different seeds still produce 18.5–28.9% adaptation. The drop in adaptation shows that models use the state, but the stronger “under-react” claim depends on absolute rates from an unvalidated judge.

Questions for authors

Please address Weaknesses 1 and 2.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mVuY

Thank you for the careful review and for recognizing the strengths of our work. We really appreciate it. We took both weaknesses seriously and ran additional experiments. Below we report the results and what we will change.

W1: Absolute JPR depends on the judge.

We agree, and this was a concern of ours as well. Table 20 shows that Gemma is the more lenient judge, which is why the paper reads JPR comparatively. To see where the gap between the judges comes from, we split JPR into its parts and compared the two judges on each (Gemma minus Qwen, in points, over 22 systems):

Average gapRange across systems
Critical invariants passed+0.3−3.4 to +5.1
Average obligation score+1.0−2.1 to +3.3
Q (partial-credit score)+1.0−2.1 to +5.0
Strict JPR+7.1+2.9 to +15.9

On average, the judges are within about a point on every part, and the large gap appears only in strict JPR. A reply passes strict JPR only if every obligation gets the top score, and that is exactly where the judges differ: on obligations where they disagree, Gemma gives the top score while Qwen gives the middle one about five times as often as the other way round (5,270 vs 1,069). Q narrows the gap but not uniformly, so we compare systems rather than absolute levels.

The rankings hold up: for 202 of the 231 system pairs, both judges order the two systems the same way and find the difference significant. No pair is significant in opposite directions. The remaining 29 pairs are all systems within four places of each other, and we will not claim an order for them.

In the revision we will:

  • Make Q, which is already in the paper, the main score, and keep strict JPR as a secondary number.
  • Report system differences as paired comparisons, mark which hold under both judges, and move Table 19 into the main text.
  • Rephrase statements such as "91.6%" and "near the ceiling" as judge-specific, so that none reads as absolute.
  • Calibrate absolute levels against a new human-labeled sample covering every system (about 25 replies per system, and more where the judges differ most).

W2: The pair judge needs a human check, and the removal control is noisy.

We agree, and we re-ran the paired study with fixed seeds, a removal condition with byte-identical prompts, a swap check that exchanges the two replies between the states (a judge that reads the replies should then fail the pair), both judges, and a blind human audit with pairs selected before any judging.

These controls also explain the noise. Two sampled replies always differ somewhat, and some of those chance differences look like the required change, so even identical prompts pass about half the time (0.53 under Gemma, 0.49 under Qwen). This baseline is not comparable to the submitted 18.5–28.9%, since the contrast sets, contracts and pair judge differ. We therefore measure adaptation against each study's own baselines, not as a raw rate. NPCBank 1.0.7 has 290 contrast groups whose contracts spell out the required change. On the 124 that change the state (4 models):

Pairs where the state must change the replyGemmaQwen
Pass rate, real pairs0.710.70
Real minus swapped pairs+0.36+0.43
Real minus state-removed pairs+0.18+0.21

Removing the state lowers adaptation by 18–21 points (22–31 in Table 60), and the swap margin indicates that the judges are reading the replies.

The human audit supports the under-reaction finding and suggests the judges are too generous. One author, blind to model, condition and judge labels, checked 48 pairs. Of the 24 that should change with the state, the author passed 9 (38%), while on the same pairs Gemma passed 14 of 23 and Qwen 12 of 22. 13 of the 15 failures were replies that did not change at all, mostly in history contrasts, where the two versions differ only in whether something was said earlier: the reply "remembers" it in both. On the audited pairs, judge–human agreement is 77% for Gemma and 83% for Qwen.

In the revision we will:

  • Report adaptation against the swap and removal baselines, under both judges.
  • Add the human audit, with a second annotator, and state the under-reaction claim for history-based state, where the controls and the human labels agree.
  • Release the new contrast set together with the audit labels.

Summary. Relative comparisons hold under both judges, and the under-reaction finding holds against controlled baselines and a blind human audit. The revision will make Q the main score, report adaptation against these baselines together with the audit, and calibrate absolute levels with human labels. We hope this addresses your concerns, and we are happy to provide further details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.