jy-crpg-bench: A Long-Horizon Benchmark for Frontier Models in an Open-World RPG

A long-horizon task requires a model to perceive its environment, decide for itself what is worth doing, and hold one plan across hours of dependent decisions. We present jy-crpg-bench, which places a frontier model inside Heroes of Jin Yong, a 1996 open-world role-playing game run under DOS emulation. Its ending, recovering the fourteen books of the Jin Yong novels, takes a first-time human player tens of hours. The game requires reading 320x200 isometric pixel art and 16-pixel Traditional Chinese, knowledge of the novels its world is built from, and a keyboard as the only tool. It has no quest log, marker or score, so the model must maintain its own map and goals, and it must win turn-based fights against the game AI. Progress is scored by milestones of the game itself and, once a model finishes, by completion time. Under a 60-minute budget, one of 11 frontier models plays past the opening of the game, and none gains experience or holds a book.