- Rating
- 1(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces jy-crpg-bench, which allows state-of-the-art models to play the unmodified 1996 Chinese RPG Heroes of Jin Yong under emulation. The models can only view the original 320×200-pixel screen and act only by pressing keys. The game has no quest log; progress is measured by 11 milestones and completion time. The authors evaluate 12 models across 42 sessions under a 60-minute budget and compare their performance against a random policy and seven publicly available human gameplay recordings.
Critical strengths
- The benchmark is engaging and challenging. Even the strongest models fail to obtain a single book, whereas four speedruns by experienced human players finish the whole game in 60 to 126 minutes.
- The scoring is reliable and hard to game. Progress is read from emulator memory, save files, and template matching on replays, none of which the models can see. Furthermore, one session that read the keypress timelines of earlier sessions was identified and excluded.
Critical weaknesses
- The evaluation does not cover the long-horizon part of the task. The paper presents jy-crpg-bench as a long-horizon benchmark, yet the 60-minute budget is sized to the opening and no session obtains a book. Models take a median of 308 keypresses to leave the starting house, mostly through trial and error, although the opening itself requires few dependent steps. The results thus leave open whether the benchmark measures long-horizon capability.
- The diagnostic conclusions lack controlled experiments. Section 5 attributes the bottlenecks to three capabilities and claims they can only come from training, yet the quantitative support is limited to cross-session rank correlations in Appendix B, and ten valid four-hour sessions in Section 4.3. No experiment isolates any of the five capabilities listed in Section 1. The evaluation also relies on a single fixed harness, although work cited in Section 3.3 notes that harness effects can exceed the differences between models.
Questions for authors
- Section 1 states that the model must utilize knowledge of the novels and set its own goals. Which of the 11 scoring milestones actually require these two capabilities? If none do, how do you support the claim that the benchmark measures long-horizon capability?
- Could you start a few additional sessions from a snapshot taken after the first battle and report whether the later milestones reveal differences between models?
- Could you run a small number of sessions with a different harness and report whether the rankings remain consistent?