acceptodds
Under review as a conference paper at ICLR 2027

RPG-AgentBench: Evaluating the Understanding–Action Gap in Long-Horizon Role-Playing

Abstract

Large language models (LLMs) can generate convincing character dialogue, but sustained role-playing also requires decisions grounded in a character's experiences, knowledge, and obligations. However, current models can lose this grounding over extended interactions. We introduce RPG-AgentBench, a benchmark built in a role-playing game environment spanning 32 worlds and 197 characters. Agents interact through dialogue and executable actions that change the world state. Three stages first assess historical understanding and its use in decisions, then test whether these abilities remain reliable in online interaction. Here, the consequences of earlier actions and new events shape the next situation, requiring models to update their understanding of the character's circumstances and decide how to act next. Experiments across five model families show that strong historical understanding does not necessarily translate into reliable decisions. Sustained interactions further reveal that models make decisions that contradict their character's stance, established facts, or prior commitments. Even Gemini-3.8-Flash, the best-performing model in our online evaluation, maintains role consistency throughout all 40 turns in only 44.8% of evaluable pressure-test trajectories. These results reveal a capability gap between expressing a character and sustaining that role: reliable role-playing requires maintaining coherence between a character's history, current circumstances, and subsequent actions throughout ongoing interaction with a player.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Role-Playing Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_geq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RPG-AgentBench, a benchmark for evaluating whether language models can sustain a role over long interactions rather than merely imitate a character’s speaking style. The benchmark uses 32 fictional worlds and 197 characters, combining long dialogue histories with executable actions whose consequences persist in a game state. It evaluates three abilities: understanding a character’s history and obligations, changing decisions when relevant past conditions change while preserving decisions under irrelevant changes, and maintaining role-consistent behavior over 40-turn online interactions. Across five model families, the paper finds a substantial gap between historical understanding and action: models often recall facts correctly but fail to revise decisions when the evidence changes, fulfill commitments, or preserve established constraints during interaction. The best tested model, Gemini-3.8-Flash, completes 44.8% of evaluable 40-turn pressure trajectories without a detected failure.

Critical strengths
  1. Existing benchmarks often focus on persona, speaking style, or factual knowledge. This paper asks whether that knowledge actually guides later decisions and actions. That is an important part of sustaining a role over time.
  2. The evaluation moves from history understanding to paired decision tests and then online interaction. This helps show where failures occur. The sensitivity tests check whether models revise decisions when relevant evidence changes. The invariance tests ask whether they remain consistent when only the wording changes.
  3. The authors check GameCore’s rule-based judgments with another LLM evaluator and human review. This does not resolve every evaluation concern, but it provides some support for the automatic failure labels. The analysis also goes beyond overall scores. It examines repeated judgments across sensitivity pairs and identifies online failures, including invalid actions, missed commitments, and violations of persistent constraints.
Critical weaknesses
  1. The 13 predefined actions make evaluation manageable, but they also limit what counts as valid role behavior. In real interactions, a character might respond through negotiation, clarification, delay, or an action outside this interface. The benchmark may therefore measure how well models follow its protocol, rather than their broader ability to sustain a role over time.
  2. The Player is generated by a model and prompted to apply pressure. The difficulty of the interaction therefore depends partly on the Player model and its prompt. The main online pass rate also excludes trajectories that end early without a confirmed failure. As the appendix notes, this can introduce selection bias, since ending the dialogue is itself a model decision.
  3. Stage 1 and parts of Stage 2 rely on an LLM grader to judge answer meaning. The alternative evaluator is also an LLM. Human review overturns some failure labels, but it does not measure missed failures or provide full precision and recall estimates. It is therefore unclear how much grading errors affect comparisons between models with similar scores.
Questions for authors
  1. How well do the results extend beyond the 13 predefined actions? Could you test situations with several reasonable choices, including those that call for negotiation, clarification, or an action outside the current interface?
  2. Could you assess evaluation reliability more fully? Human review identifies some false positives, but what about missed failures?
  3. Can the benchmark help improve agents, not just identify failures? For example, could memory, planning, state tracking, or action-feedback training improve Stage 2 sensitivity and the 40-turn pass rate?
Reviewer_Dz59OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RPG-AgentBench evaluates history-grounded role-playing across 32 fictional worlds and 197 characters. It combines dialogue with executable actions and persistent state, using three stages: comprehension of 600-round histories, paired decisions under relevant and irrelevant historical changes, and online interaction lasting up to 40 turns. Across five model families, the authors report substantial historical-sensitivity and commitment-fulfillment failures, with a maximum 40-turn pass rate of 44.8% among evaluable trajectories. Evaluation reliability is examined through an alternative LLM evaluator and human review.

Critical strengths
  1. Executable consequences make behavioral consistency measurable.
    The framework distinguishes verbal statements, executed actions, and fulfilled commitments. It evaluates both maintaining boundaries and acting when obligations become due, making blanket refusal insufficient. This provides concrete signals for assessing sustained, consequential role behavior.

  2. Controlled historical perturbations offer useful diagnostics.
    Sensitivity and invariance pairs test whether decisions respond appropriately to historical changes. The finding that 48.7% of claim-pair observations repeat the same judgment despite requiring opposite answers provides substantive evidence of historical insensitivity (Section 5.3). Detailed action specifications, prompts, and supplementary validation further support the benchmark’s utility.

Critical weaknesses
  1. Online comparisons confound NPC reliability with Player strength and completion selection.
    Each candidate supplies its own adversarial Player (Appendix A), so models face different adaptive pressure. Additionally, early end_dialogue calls exclude roughly one-fifth of Qwen and Kimi trajectories from the headline scores (Table 7). The authors transparently acknowledge this issue and report a consistent ranking on a common subset, but that analysis does not control Player difficulty or eliminate behavior-dependent selection. Consequently, the ranking reflects NPC–Player pairings conditional on evaluability.
Questions for authors
  1. Do online rankings persist with a shared Player or a balanced Player–NPC cross-play evaluation, using a predefined early-termination policy and reporting complete-episode success over all attempted trajectories?
  2. Can the authors report decision accuracy conditional on correctly identifying the same decision-relevant historical facts, together with per-branch accuracy? Do failures persist when the relevant facts or current-state summary are supplied directly?
Reviewer_qXLMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RPG-AgentBench, a benchmark for evaluating the gap between historical understanding and decision-making in long-horizon role-playing agents. The benchmark covers 32 fictional worlds and 197 characters, with three evaluation stages: (1) understanding character histories, (2) making decisions under controlled history variations, and (3) maintaining role consistency through 40-turn interactions in an executable game environment. The framework uses rule-based state checks to evaluate whether agents' actions respect established facts, commitments, and changing circumstances. Experiments across five LLMs show that strong historical understanding does not necessarily translate into reliable decision-making or sustained role consistency.

Critical strengths

not sure.

Critical weaknesses
  • The motivation for proposing this benchmark is not clear throughout the paper. If models improve their performance on this benchmark, what specific abilities are we expecting to improve? The introduction may also need to be rewritten to better explain the motivation.

  • I think I have seen several similar benchmarks before. As shown in Table 1, this benchmark adds rule-based state checks and controlled history tests as new dimensions compared to prior benchmarks, but why are these additions critical? It seems that these dimensions could be added to existing benchmarks relatively easily as well.

  • Overall, the paper feels like it was written using AI, and the key points: why this benchmark is needed, what model abilities it measures, and the motivations behind each experiment are not conveyed clearly enough.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.