- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RPG-AgentBench, a benchmark for evaluating whether language models can sustain a role over long interactions rather than merely imitate a character’s speaking style. The benchmark uses 32 fictional worlds and 197 characters, combining long dialogue histories with executable actions whose consequences persist in a game state. It evaluates three abilities: understanding a character’s history and obligations, changing decisions when relevant past conditions change while preserving decisions under irrelevant changes, and maintaining role-consistent behavior over 40-turn online interactions. Across five model families, the paper finds a substantial gap between historical understanding and action: models often recall facts correctly but fail to revise decisions when the evidence changes, fulfill commitments, or preserve established constraints during interaction. The best tested model, Gemini-3.8-Flash, completes 44.8% of evaluable 40-turn pressure trajectories without a detected failure.
Critical strengths
- Existing benchmarks often focus on persona, speaking style, or factual knowledge. This paper asks whether that knowledge actually guides later decisions and actions. That is an important part of sustaining a role over time.
- The evaluation moves from history understanding to paired decision tests and then online interaction. This helps show where failures occur. The sensitivity tests check whether models revise decisions when relevant evidence changes. The invariance tests ask whether they remain consistent when only the wording changes.
- The authors check GameCore’s rule-based judgments with another LLM evaluator and human review. This does not resolve every evaluation concern, but it provides some support for the automatic failure labels. The analysis also goes beyond overall scores. It examines repeated judgments across sensitivity pairs and identifies online failures, including invalid actions, missed commitments, and violations of persistent constraints.
Critical weaknesses
- The 13 predefined actions make evaluation manageable, but they also limit what counts as valid role behavior. In real interactions, a character might respond through negotiation, clarification, delay, or an action outside this interface. The benchmark may therefore measure how well models follow its protocol, rather than their broader ability to sustain a role over time.
- The Player is generated by a model and prompted to apply pressure. The difficulty of the interaction therefore depends partly on the Player model and its prompt. The main online pass rate also excludes trajectories that end early without a confirmed failure. As the appendix notes, this can introduce selection bias, since ending the dialogue is itself a model decision.
- Stage 1 and parts of Stage 2 rely on an LLM grader to judge answer meaning. The alternative evaluator is also an LLM. Human review overturns some failure labels, but it does not measure missed failures or provide full precision and recall estimates. It is therefore unclear how much grading errors affect comparisons between models with similar scores.
Questions for authors
- How well do the results extend beyond the 13 predefined actions? Could you test situations with several reasonable choices, including those that call for negotiation, clarification, or an action outside the current interface?
- Could you assess evaluation reliability more fully? Human review identifies some false positives, but what about missed failures?
- Can the benchmark help improve agents, not just identify failures? For example, could memory, planning, state tracking, or action-feedback training improve Stage 2 sensitivity and the 40-turn pass rate?