- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GlyphBench, which is an environment suite for RL post-training of language-model agents. It has 362 tasks across nine game families (Classics, MiniGrid, MiniHack, Craftax, MiniAtari, Procgen, plus extended Atari, full Craftax and NetHack). All of them render spatial state as 2D Unicode grids with a legend and HUD, and use named discrete actions. The 303 standard tasks are reimplemented from scratch, with episodes capped at 512 steps and returns bounded to [-1, 1] so tasks can be mixed. The paper uses the suite in three ways. As a benchmark: zero-shot evaluation of Qwen models, and a Craftax study showing glyph observations beat native text and pixels (e.g., 11.4% vs 0.3% vs 3.7% of max reward for one model). As a research tool: a learning-rate × group-size sweep on Snake under a fixed rollout budget. As a post-training suite: one Qwen3.5-4B policy trained on 100 tasks improves on held-out seeds (0.012 → 0.319 return) and reaches 63.5% on Reasoning Gym, ahead of math-RL (62.3%), code-RL (59.0%) and the base model (56.0%).
The framework is a useful contribution to the community, but without code reveal, this makes it hard to better understand how the framework could be used.
Critical strengths
- GlyphBench cover broad, well-designed environment suite for LLM RL. Bringing many established RL game families into one spatial text format, with bounded episodes and returns, makes it practical to train and compare across tasks. The glyph format is well motivated, and the Craftax comparison shows it matters.
- Careful experiments that show what the suite can be used for. The optimization sweep, multitask training with held-out seeds, and the transfer study use paired confidence intervals and are reported honestly. This includes checking non-game Reasoning Gym tasks separately and stating plainly that MathArena and SWE-bench gains are inconclusive.
Critical weaknesses
-
No code for a paper whose main contribution is a framework. The paper's value lies mostly in the environments and the interface, but no code, anonymous repository or supplementary material is provided. The interface is described in only two sentences (Appendix B.1: Gymnasium convention plus a Verifiers adapter). There's no API description, no observation/action schema beyond the Craftax example, no detail on action parsing, and no guidance on using the suite outside Prime-RL (e.g., with verl or slime). This matters even more because the 303 standard tasks were reimplemented from scratch, partly with coding agents (per the AI use statement), so reviewers cannot check whether the game mechanics are correct. For the community to benefit, people need to see how to plug these environments into their own training and evaluation pipelines. Until the code is available, I can't fully assess the main contribution, and I would raise my score once it is.
-
Missing closely related game platforms. Two of the most directly comparable LLM game frameworks are not cited or discussed: TextArena [1], a collection of 57+ text games with a Gym-like API for evaluating and training LLMs, and KORGym [2], a Gymnasium-based platform with 50+ text and visual games and RL settings. The paper should explain how GlyphBench differs, for example in the spatial glyph format, the focus on RL post-training, or the scale of single-player RL tasks, so readers understand what it adds.
[1] Guertler et al. TextArena. https://arxiv.org/abs/2504.11442
[2] Shi et al. KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation. https://arxiv.org/abs/2505.14552
Questions for authors
- Can you share an anonymous repository or supplementary materials during the rebuttal? Concretely: what does an environment's API look like (observation and action types, how a model response is parsed into an action), and how would someone use GlyphBench with a trainer other than Prime-RL?
- Since the standard tasks were reimplemented from scratch with help from coding agents, how did you verify that their mechanics and rewards match the original games? For example, are there tests against the original engines, or a manual audit?
- How does GlyphBench compare with TextArena and KORGym in task coverage, interface and training support? Would including some of their tasks change any conclusions?
2 comments
Thanks for the author's response. The release of the framework makes the paper's contribution and interface more clear; although I still have some concerns on how this work diff from existing Gym works, porting them for RL has already been done by the Prime Intellect Environment Hub, the paper's contribution is still valid. Therefore, I have raised the score. I would still suggest the author to spend more effort on discussing how to differentiate this work from existing gym work.
We have attached the full anonymized GlyphBench source code as supplementary material with this rebuttal. The ZIP includes the environments, training and evaluation code, tests, and guide to adding new games. Appendix B.1 shows the environment API. reset(seed) returns the text observation and metadata. The observation contains the glyph grid, legend, and any HUD or message. The model returns a named action such as
<action>MOVE_FORWARD</action>. The adapter converts it to an integer and calls step(action), which returns the next observation, reward, termination and truncation flags, and metadata.Another trainer can use the same reset/step loop to collect observations, model outputs, actions, and rewards, then pass those trajectories to its optimizer. Our Verifiers adapter uses this interface for multi-turn training and evaluation with Prime-RL (Appendix B.1).
TextArena emphasizes interaction with other players, including negotiation and deception. GlyphBench emphasizes interaction with a world. In MiniHack, for example, the agent has to locate itself, navigate around obstacles, and use what it saw on earlier turns to reach a goal. KORGym overlaps with GlyphBench on spatial games such as mazes, Sokoban, and Snake. Both KORGym and TextArena support multi-turn play and training. What GlyphBench adds is a common spatial text interface and the RL setup we used to train an LLM on any one game or on a mixture of 100 tasks (§2 and Appendix A.1).
We added KORGym’s LongCat and four-peg Tower of Hanoi to Classics. KORGym scores a complete move sequence for these two games. In GlyphBench, the model makes one move, sees the updated board, and chooses its next move. On the same 25 seeds per game, Qwen3.8-27B solved 21 LongCat layouts and all 25 Hanoi puzzles. Qwen3.5-4B solved 9 and 4, respectively. Tokens include the prompt and completion at each turn. Both averages include all 25 episodes (Appendix B.2, Table 2).
The larger model also used fewer turns and tokens on both games. These additions extend the model comparison to two new tasks. The transfer results in §5.3 use the original 100-task training mixture. The contribution guide shows how to add another game in the same way so it can use the existing evaluation and RL code (Appendix B.2).
For full Craftax and NetHack, the original engines compute the next state and reward. For the standard tasks, we reimplemented the mechanics and bounded the rewards for training. We test specific rules directly. For example, picking up the key in MiniHack Corridor must earn its reward only once. Replaying the same actions on the same Procgen seed checks that our implementation is deterministic. These tests check our rules and rewards. They do not establish that every trajectory matches the original engine.
For the two games we adapted from KORGym, we could check against the source directly. We generated 100 solutions each for LongCat and four-peg Tower of Hanoi and passed the move sequences to KORGym’s own verifiers. They accepted all 200. We also replay known solutions in GlyphBench and check that each game gives its success reward exactly once (Appendix B.2).