acceptodds
Under review as a conference paper at ICLR 2027

jy-crpg-bench: A Long-Horizon Benchmark for Frontier Models in an Open-World RPG

Abstract

A long-horizon task requires a model to perceive its environment, decide for itself what is worth doing, and hold one plan across hours of dependent decisions. We present jy-crpg-bench, which places a frontier model inside Heroes of Jin Yong, a 1996 open-world role-playing game run under DOS emulation. Its ending, recovering the fourteen books of the Jin Yong novels, takes a first-time human player tens of hours. The game requires reading 320x200 isometric pixel art and 16-pixel Traditional Chinese, knowledge of the novels its world is built from, and a keyboard as the only tool. It has no quest log, marker or score, so the model must maintain its own map and goals, and it must win turn-based fights against the game AI. Progress is scored by milestones of the game itself and, once a model finishes, by completion time. Under a 60-minute budget, one of 11 frontier models plays past the opening of the game, and none gains experience or holds a book.

Primary area:
datasets and benchmarks
Keywords:
long-horizon agents, multimodal agents, benchmark, vision-language models, video games, planning, open-world, self-directed evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8vhDOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces jy-crpg-bench, which allows state-of-the-art models to play the unmodified 1996 Chinese RPG Heroes of Jin Yong under emulation. The models can only view the original 320×200-pixel screen and act only by pressing keys. The game has no quest log; progress is measured by 11 milestones and completion time. The authors evaluate 12 models across 42 sessions under a 60-minute budget and compare their performance against a random policy and seven publicly available human gameplay recordings.

Critical strengths
  1. The benchmark is engaging and challenging. Even the strongest models fail to obtain a single book, whereas four speedruns by experienced human players finish the whole game in 60 to 126 minutes.
  2. The scoring is reliable and hard to game. Progress is read from emulator memory, save files, and template matching on replays, none of which the models can see. Furthermore, one session that read the keypress timelines of earlier sessions was identified and excluded.
Critical weaknesses
  1. The evaluation does not cover the long-horizon part of the task. The paper presents jy-crpg-bench as a long-horizon benchmark, yet the 60-minute budget is sized to the opening and no session obtains a book. Models take a median of 308 keypresses to leave the starting house, mostly through trial and error, although the opening itself requires few dependent steps. The results thus leave open whether the benchmark measures long-horizon capability.
  2. The diagnostic conclusions lack controlled experiments. Section 5 attributes the bottlenecks to three capabilities and claims they can only come from training, yet the quantitative support is limited to cross-session rank correlations in Appendix B, and ten valid four-hour sessions in Section 4.3. No experiment isolates any of the five capabilities listed in Section 1. The evaluation also relies on a single fixed harness, although work cited in Section 3.3 notes that harness effects can exceed the differences between models.
Questions for authors
  1. Section 1 states that the model must utilize knowledge of the novels and set its own goals. Which of the 11 scoring milestones actually require these two capabilities? If none do, how do you support the claim that the benchmark measures long-horizon capability?
  2. Could you start a few additional sessions from a snapshot taken after the first battle and report whether the later milestones reveal differences between models?
  3. Could you run a small number of sessions with a different harness and report whether the rankings remain consistent?
Reviewer_4T2VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors build a role-play open-world benchmark based on the 1996 Chinese RPG Heroes of Jin Yong. A frontier model is asked to play the game from raw screen pixels and keyboard input only, with no game-state access, no map, and no memory scaffold. This serves as a long-horizon planning context for agents and models are evaluated by 11 in-game milestones and completion time. Across 12 models and 42 1-hour sessions, the best model reaches 10 of 11 milestones once; no session obtains a book; longer 4-hour sessions get no further. This is compared with a human with a book, which completes within 105 mins.

Critical strengths
  1. This topic is important and matters. An open-world, long-horizon agent benchmark with a real game essentially simulates human play, which is harder and more realistic than most existing game benchmarks.

  2. Milestones are defined and read from emulator memory and saves, and the numbers are reproducible from committed code and results.

Critical weaknesses
  1. The paper is generally clear but very dense. Sometimes the main text is hard to follow on its own.
  2. Language is mixed. The game, where on-screen text and the instructions are all in Chinese, and the paper claims that by knowing the Jin Yong novels, it can help. Models with more Chinese pretraining may be favoured in that sense. Also, acting on what a character says in dialogue could be a reading failure rather than a planning failure, e.g., during hermit-dialogue filter turning what the model reads into an action.
  3. The contribution of the work is long-horizon planning, but all observed failures occur in the opening segment, and 4-hour sessions do not progress beyond 1-hour ones. This suggests current difficulty is dominated by perception and low-level control rather than planning. Controlled variants would let the authors attribute failures to specific capabilities and substantiate the long-horizon claim. The action set is also small (9 keys), so the openness is in the world, not the actions.
  4. The ranking rests on very little. No session finishes, so ranking by completion time is empty. Each model is credited with the best milestone any of its sessions reached, with only 2 to 6 sessions per model. The gap between the top model (10 milestones, in one of two sessions) and the next (8) is one run. There is no variance reported and no way to tell if models are actually distinguishable.
Questions for authors

Besides the above, as the game keeps running while the model thinks, is that underestimating a slow but capable model? Is it possible to compare under the same action budget or milestones?

Reviewer_C7kJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces JY-CRPG-Bench, where models play the original Heroes of Jin Yong using screenshots and keyboard commands. Traditional Chinese text, isometric navigation, exploration, and turn-based combat make it quite different from the settings in most agent benchmarks. The authors have also put considerable care into measuring progress through game state, saves, and replay events. The main study covers 12 models with two to six one-hour sessions each, a random policy, and seven published human videos. Ten four-hour sessions from nine models are analyzed separately.

Critical strengths
  1. The environment is a good addition to agent evaluation. Small Traditional Chinese text, isometric movement, sparse guidance, and persistent consequences require several capabilities to work together. It is useful to see an evaluation setting outside the usual English-language interfaces.

  2. The progress measurements are well documented. Section 3.4 and Appendix D explain which records support each milestone, cross-check saves and memory against replay events, and manually inspect the first detected template events. This gives me more confidence in the scores than an assessment based on what the agent says it accomplished.

Critical weaknesses

My main concern is that it is still unclear how much of the difficulty is long-horizon reasoning. Most agents stop making progress at navigation or dialogue, and none obtains a book (Sections 4.1–4.3). These failures could come from reading small glyphs, grounding movement, or lacking game knowledge, as well as planning or memory. Although agents can read and write their own files, the study does not test a common structured-notes scaffold for locations, dialogue facts, goals, and failed actions. The current evaluation therefore does not separate interface bottlenecks from failures that might be addressed by more systematic memory and goal tracking.

Questions for authors
  • I'm curious whether a simple structured notebook would change the result: visited locations, dialogue facts, current goals, prerequisites, and failed actions, updated after each meaningful interaction. Agents can already use files, but do the traces show whether they maintain and consult useful notes? A comparison of raw observations with and without this scaffold, followed by the same notes with reliable text/state or coordinate observations, would help locate the bottleneck. Have you also tried larger images, a known route, or a later campaign checkpoint as diagnostic settings?

  • Do you have results for first-time human players using the same instructions, or comparisons under a fixed action budget? These would help interpret the gap to experienced players and the effect of different model latencies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.