acceptodds
Under review as a conference paper at ICLR 2027

GlyphBench: A Playground for Language-Model Reinforcement Learning

Abstract

We introduce GlyphBench, an environment suite for reinforcement learning (RL) post-training of language-model agents, with over 360 tasks spanning diverse games. GlyphBench renders spatial observations as two-dimensional Unicode grids and connects training, evaluation, and trajectory replay through a unified interface designed to support efficient and reproducible research. We use GlyphBench to study how observation interfaces, reasoning effort, and agent harnesses affect performance, and how RL configurations shape learning dynamics. Our results show that glyph observations outperform native text and pixels in our Craftax experiments, with further gains on several BALROG environments. RL on 100 GlyphBench tasks improves Qwen3.5-4B on held-out Reasoning Gym problems, reaching 63.48% accuracy and outperforming the base model, a math-trained baseline, and a code-trained baseline. These experiments provide empirical evidence that reasoning gains from gameplay can yield stronger transfer than math or code. Together, these results highlight GlyphBench's value as a testbed for systematic research on how language-model agents learn, interact, and generalize.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, language models, benchmark, AI agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XYTwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GlyphBench, which is an environment suite for RL post-training of language-model agents. It has 362 tasks across nine game families (Classics, MiniGrid, MiniHack, Craftax, MiniAtari, Procgen, plus extended Atari, full Craftax and NetHack). All of them render spatial state as 2D Unicode grids with a legend and HUD, and use named discrete actions. The 303 standard tasks are reimplemented from scratch, with episodes capped at 512 steps and returns bounded to [-1, 1] so tasks can be mixed. The paper uses the suite in three ways. As a benchmark: zero-shot evaluation of Qwen models, and a Craftax study showing glyph observations beat native text and pixels (e.g., 11.4% vs 0.3% vs 3.7% of max reward for one model). As a research tool: a learning-rate × group-size sweep on Snake under a fixed rollout budget. As a post-training suite: one Qwen3.5-4B policy trained on 100 tasks improves on held-out seeds (0.012 → 0.319 return) and reaches 63.5% on Reasoning Gym, ahead of math-RL (62.3%), code-RL (59.0%) and the base model (56.0%).

The framework is a useful contribution to the community, but without code reveal, this makes it hard to better understand how the framework could be used.

Critical strengths
  1. GlyphBench cover broad, well-designed environment suite for LLM RL. Bringing many established RL game families into one spatial text format, with bounded episodes and returns, makes it practical to train and compare across tasks. The glyph format is well motivated, and the Craftax comparison shows it matters.
  2. Careful experiments that show what the suite can be used for. The optimization sweep, multitask training with held-out seeds, and the transfer study use paired confidence intervals and are reported honestly. This includes checking non-game Reasoning Gym tasks separately and stating plainly that MathArena and SWE-bench gains are inconclusive.
Critical weaknesses
  1. No code for a paper whose main contribution is a framework. The paper's value lies mostly in the environments and the interface, but no code, anonymous repository or supplementary material is provided. The interface is described in only two sentences (Appendix B.1: Gymnasium convention plus a Verifiers adapter). There's no API description, no observation/action schema beyond the Craftax example, no detail on action parsing, and no guidance on using the suite outside Prime-RL (e.g., with verl or slime). This matters even more because the 303 standard tasks were reimplemented from scratch, partly with coding agents (per the AI use statement), so reviewers cannot check whether the game mechanics are correct. For the community to benefit, people need to see how to plug these environments into their own training and evaluation pipelines. Until the code is available, I can't fully assess the main contribution, and I would raise my score once it is.

  2. Missing closely related game platforms. Two of the most directly comparable LLM game frameworks are not cited or discussed: TextArena [1], a collection of 57+ text games with a Gym-like API for evaluating and training LLMs, and KORGym [2], a Gymnasium-based platform with 50+ text and visual games and RL settings. The paper should explain how GlyphBench differs, for example in the spatial glyph format, the focus on RL post-training, or the scale of single-player RL tasks, so readers understand what it adds.

[1] Guertler et al. TextArena. https://arxiv.org/abs/2504.11442

[2] Shi et al. KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation. https://arxiv.org/abs/2505.14552

Questions for authors
  1. Can you share an anonymous repository or supplementary materials during the rebuttal? Concretely: what does an environment's API look like (observation and action types, how a model response is parsed into an action), and how would someone use GlyphBench with a trainer other than Prime-RL?
  2. Since the standard tasks were reimplemented from scratch with help from coding agents, how did you verify that their mechanics and rewards match the original games? For example, are there tests against the original engines, or a manual audit?
  3. How does GlyphBench compare with TextArena and KORGym in task coverage, interface and training support? Would including some of their tasks change any conclusions?
2 comments

“Can you share an anonymous repository or supplementary materials during the rebuttal? Concretely: what does an environment's API look like (observation and action types, how a model response is parsed into an action), and how would someone use GlyphBench with a trainer other than Prime-RL?”

We have attached the full anonymized GlyphBench source code as supplementary material with this rebuttal. The ZIP includes the environments, training and evaluation code, tests, and guide to adding new games. Appendix B.1 shows the environment API. reset(seed) returns the text observation and metadata. The observation contains the glyph grid, legend, and any HUD or message. The model returns a named action such as <action>MOVE_FORWARD</action>. The adapter converts it to an integer and calls step(action), which returns the next observation, reward, termination and truncation flags, and metadata.

Another trainer can use the same reset/step loop to collect observations, model outputs, actions, and rewards, then pass those trajectories to its optimizer. Our Verifiers adapter uses this interface for multi-turn training and evaluation with Prime-RL (Appendix B.1).

“How does GlyphBench compare with TextArena and KORGym in task coverage, interface and training support? Would including some of their tasks change any conclusions?”

TextArena emphasizes interaction with other players, including negotiation and deception. GlyphBench emphasizes interaction with a world. In MiniHack, for example, the agent has to locate itself, navigate around obstacles, and use what it saw on earlier turns to reach a goal. KORGym overlaps with GlyphBench on spatial games such as mazes, Sokoban, and Snake. Both KORGym and TextArena support multi-turn play and training. What GlyphBench adds is a common spatial text interface and the RL setup we used to train an LLM on any one game or on a mixture of 100 tasks (§2 and Appendix A.1).

We added KORGym’s LongCat and four-peg Tower of Hanoi to Classics. KORGym scores a complete move sequence for these two games. In GlyphBench, the model makes one move, sees the updated board, and chooses its next move. On the same 25 seeds per game, Qwen3.8-27B solved 21 LongCat layouts and all 25 Hanoi puzzles. Qwen3.5-4B solved 9 and 4, respectively. Tokens include the prompt and completion at each turn. Both averages include all 25 episodes (Appendix B.2, Table 2).

GameModelSuccessTokens / episodeTurns / episode
LongCatQwen3.5-4B36% (9/25)757,30025.3
LongCatQwen3.8-27B84% (21/25)61,5127.4
Tower of HanoiQwen3.5-4B16% (4/25)1,419,54641.2
Tower of HanoiQwen3.8-27B100% (25/25)235,57713.4

The larger model also used fewer turns and tokens on both games. These additions extend the model comparison to two new tasks. The transfer results in §5.3 use the original 100-task training mixture. The contribution guide shows how to add another game in the same way so it can use the existing evaluation and RL code (Appendix B.2).

“Since the standard tasks were reimplemented from scratch with help from coding agents, how did you verify that their mechanics and rewards match the original games? For example, are there tests against the original engines, or a manual audit?”

For full Craftax and NetHack, the original engines compute the next state and reward. For the standard tasks, we reimplemented the mechanics and bounded the rewards for training. We test specific rules directly. For example, picking up the key in MiniHack Corridor must earn its reward only once. Replaying the same actions on the same Procgen seed checks that our implementation is deterministic. These tests check our rules and rewards. They do not establish that every trajectory matches the original engine.

For the two games we adapted from KORGym, we could check against the source directly. We generated 100 solutions each for LongCat and four-peg Tower of Hanoi and passed the move sequences to KORGym’s own verifiers. They accepted all 200. We also replay known solutions in GlyphBench and check that each game gives its success reward exactly once (Appendix B.2).

Thanks for the author's response. The release of the framework makes the paper's contribution and interface more clear; although I still have some concerns on how this work diff from existing Gym works, porting them for RL has already been done by the Prime Intellect Environment Hub, the paper's contribution is still valid. Therefore, I have raised the score. I would still suggest the author to spend more effort on discussing how to differentiate this work from existing gym work.

Reviewer_2gSROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GlyphBench, which supplies game states to language models as grids of text symbols, legends, and status information, and accepts named actions in return. It provides a collection of puzzles, arcade-style tasks, and adapted reinforcement-learning games for evaluation and training. The authors argue that such interaction can exercise abilities that single-answer problems do not, but differences between game interfaces make broad training studies cumbersome. GlyphBench supplies a common format so the same agent and training tools can operate across varied games, and investigates whether learning through gameplay improves performance on separate reasoning tasks. The practical idea is to present a game as a grid of text symbols, a legend, and status information, with named actions the model can issue. A shared interface handles running the game, collecting experience, training from it, and replaying interactions for inspection. The collection spans puzzles, arcade-style tasks, and adaptations of established reinforcement-learning games. Standard tasks simplify and bound objectives; an extended track includes longer tasks and wrappers around the original Craftax and NetHack games. Success on a simplified adaptation is not mastery of the original game.

The contribution is a game collection with shared tools for evaluation, training on individual tasks or mixtures, and replaying interactions. Shared interfaces, spatial text, and learning from games are established ideas. The contribution is the integrated collection and demonstrated research uses, rather than a new optimization algorithm. Studies test how presentation and agent support affect results. Here the agent harness means the prompts, retained history, notes, and tool logic surrounding the model. A spatial-text Craftax configuration improves mean performance over the tested image and original-text configurations. However, the observation formats expose different state information, so the experiment does not isolate text layout as the cause of the improvement.

Evaluation covers the standard suite, while training studies use individual tasks and a multitask subset. Average reward across the multitask mixture improves on new game instances generated from held-out seeds. This shows that the shared training setup produces learning beyond the exact game instances used in training. It does not show improvement on every task or generalization to unseen game families. Snake experiments vary learning rate and the size of groups of sampled episodes used for updates. With a fixed total episode budget, smaller groups allow more updates; these experiments test that allocation choice rather than group size alone. Gameplay-, mathematics-, and code-trained versions of the same base model answer identical Reasoning Gym test problems, a collection of generated reasoning tasks. At the largest allowed response length, gameplay training improves per-attempt accuracy over the base and narrowly exceeds the mathematics-trained model, called Math RL. In a separate evaluation with a shorter allowed response, improvement over the base model also holds after excluding Reasoning Gym's game tasks. Changes on competition mathematics (MathArena) and software issue resolution (SWE-bench) are inconclusive. Math RL has higher per-attempt accuracy at the shortest tested response limits, while the difference is inconclusive at an intermediate limit. At the largest limit, its observed advantage in success across multiple attempts is also inconclusive. Uncertainty estimates resample shared test problems while keeping each problem's attempts together; they do not measure variation across separately trained models. The evaluated training checkpoints and their cumulative generated training tokens remain unspecified. Overall, the experiments support a usable learning resource and transfer to the tested reasoning problems more strongly than a general claim that game training is better than mathematics or code training.

Critical strengths
  1. The shared interface supports an exercised research workflow: evaluate varied games, train on an individual task or mixture, and inspect recorded interactions. Single-task learning and improved mean reward on new instances of the multitask mixture show that the integration is usable, beyond merely listing environments. This supports its value as a training resource.

  2. Gameplay training improves performance on Reasoning Gym relative to the base model before this additional training, including when that benchmark's game tasks are excluded. Comparisons use the same test problems and account for repeated attempts in their evaluation uncertainty. The authors also report inconclusive transfer to competition mathematics and software tasks, and acknowledge that the comparison with Math RL depends on response length and whether success is measured per attempt or across attempts. This is useful evidence of reasoning transfer with explicit limits.

Critical weaknesses

Using games to evaluate and train language-model agents is already established. GlyphBench brings varied games into a shared interface, but I am not convinced that the paper demonstrates a sufficiently important research need that existing resources cannot meet. The experiments show that agents can learn these tasks and that gameplay training improves performance on some separate reasoning problems. They establish that the collection works, but leave unclear what new question it enables researchers to answer, or what substantial obstacle it removes compared with existing game-based training resources. My main concern is therefore the collection’s incremental research value.

The external test procedure is described, but the trained models being tested are not fully identified. For the game-, math-, and code-trained conditions, the paper does not clearly state which saved training checkpoint produced the reported results, how it was chosen, or its cumulative generated training tokens. This prevents reconstructing the comparison and assessing whether differences in training exposure or model selection contributed to the ranking.

Questions for authors

The shared infrastructure and training experiments represent substantial work. I would encourage the authors to use this resource to develop a sharper scientific finding. In particular, the dependence of the gameplay-versus-mathematics comparison on response allowance and evaluation metric is interesting. Identifying which reasoning tasks account for these differences, and testing an explanation for them, would make the paper more informative than aggregate performance comparisons alone.

2 comments

“For the game-, math-, and code-trained conditions, the paper does not clearly state which saved training checkpoint produced the reported results, how it was chosen, or its cumulative generated training tokens.”

We use the checkpoint after 100 gradient steps. Appendix I.1 now shows the learning curves, rollout counts, and output tokens. On the second sets of 100 problems, held-out math accuracy rose from 54.7% to 96.1% and held-out code accuracy rose from 15.6% to 38.3%. The math, code, and game runs generated 1.207B, 2.126B, and 2.534B output tokens, respectively. On Reasoning Gym at 32K, they reach 60.71%, 61.62%, and 63.48%.

“In particular, the dependence of the gameplay-versus-mathematics comparison on response allowance and evaluation metric is interesting. Identifying which reasoning tasks account for these differences, and testing an explanation for them, would make the paper more informative than aggregate performance comparisons alone.”

Games and graphs account for most of the 32K advantage over the first math policy. They contribute 0.89 and 0.71 points to overall accuracy. Algebra and ARC contribute another 0.33 and 0.13 points. Geometry and cognition favor Math RL and reduce the gap by 0.55 and 0.32 points.

Both math policies score higher than GlyphBench RL at 2K and 8K. At 32K, GlyphBench RL scores higher than either math policy on algebra, ARC, games, and graphs. Both math policies score higher on cognition and geometry. Appendix I.3 shows how each category contributes to the 1.19-point difference from the original math run. The table below gives accuracy (%) for both math training sets on the same problems.

CategoryGlyphBench RLMath RL (1)Math RL (2)
Algebra79.5677.8376.72
ARC55.7854.4450.56
Cognition40.5241.9540.57
Games61.4658.0057.42
Geometry53.6762.1755.33
Graphs88.1383.7383.20

“The experiments show that agents can learn these tasks and that gameplay training improves performance on some separate reasoning problems. They establish that the collection works, but leave unclear what new question it enables researchers to answer, or what substantial obstacle it removes compared with existing game-based training resources.”

Does learning to navigate in one game help an LLM find its way in another? For example, we could first measure how often the base model finds a goal in MiniHack. We could then train two copies of that model for the same number of episodes, one on MiniGrid navigation tasks and one on a mixture of other games. Testing both on the same unseen MiniHack layouts would tell us whether navigation training transfers better than the comparison mixture. lmgame-Bench studied transfer from individual games. GlyphBench lets us vary the whole training mixture while keeping the observation format, rollout code, and trainer fixed. The paper uses this same setup to benchmark agents (§5.1), study how they learn (§5.2), and train one policy across 100 tasks (§5.3).

We have made our best effort to address each of your concerns, including clarifying the checkpoint selection and training budgets, identifying which tasks drive the transfer gains, and explaining what research questions GlyphBench enables. We would be happy to clarify any remaining points.

We respectfully ask that you reconsider your rating in light of our rebuttal.

Reviewer_pYftOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GlyphBench, a collection of 362 game tasks with a shared interface for language-model training, evaluation, and trajectory inspection. Observations use two-dimensional Unicode grids to preserve spatial structure. The experiments study observation formats, agent harnesses, and RL training configurations. Training Qwen3.5-4B on 100 tasks improves performance on new instances of those tasks and raises Reasoning Gym accuracy from 56.04% to 63.48%, compared with 62.29% for the math-trained baseline and 58.98% for the code-trained baseline.

Critical strengths
  1. The suite could be a useful research resource. Bringing diverse games into a consistent interface, with bounded episodes, standardized returns, and trajectory replay, addresses practical difficulties in studying multi-turn RL. The combination of evaluation and training support is valuable, particularly for researchers who otherwise need to maintain separate environment integrations.

  2. The experiments demonstrate useful learning and transfer. The improvement on Reasoning Gym is substantial relative to the starting model, and the evaluation uses identical problems across policies with paired uncertainty estimates. The observation-format experiments also provide practical evidence that interface design can strongly affect agent performance. These results give the suite a credible use beyond simply collecting more games.

Critical weaknesses
  1. The additional contribution over existing benchmarks is not sufficiently established. Unified game environments, spatial text observations, and transfer from game training all have close precedents. For example, Agentick (https://arxiv.org/abs/2605.06869) already provides a common interface with multiple observation formats and reports advantages of ASCII observations, while lmgame-Bench (https://arxiv.org/abs/2505.15146v1) demonstrates transfer from game RL to other games and external planning tasks. GlyphBench expands the collection considerably, but I am less convinced that the paper establishes what this expansion enables scientifically. A resource paper does not need a new learning algorithm, but it should demonstrate why its coverage or design supports investigations that existing resources cannot adequately support

  2. The comparison with math and code training does not isolate the value of gameplay. Section 5.3 compares 100 game tasks against sets of 100 math and 100 code problems. These are different units of training diversity: a game task can generate many distinct situations and trajectories. Using the same optimization recipe does not establish comparable data diversity or computational cost. The 1.19-point advantage over Math RL is statistically supported for the evaluated checkpoints, but the reported intervals do not establish robustness across training runs. Together with the inconclusive results on MathArena and SWE-bench, this makes the broader transfer claim less persuasive than the improvement over the base model.

Questions for authors
  1. Clarification: What specific capability coverage or experimental control does GlyphBench add beyond the closest existing suites? Which results demonstrate the importance of those additions beyond increasing the number of tasks?

  2. Clarification: How do the game, math, and code training runs compare in distinct training instances, total generated tokens, optimization steps, and checkpoint-selection criteria? These quantities are needed to interpret the transfer comparison.

  3. Evidence needed for the current claim: Does the advantage over Math RL persist with more diverse math training data, comparable computational budgets, and independent training seeds? This would materially affect my assessment of the claim that gameplay provides stronger reasoning transfer.

2 comments

“Evidence needed for the current claim: Does the advantage over Math RL persist with more diverse math training data, comparable computational budgets, and independent training seeds? This would materially affect my assessment of the claim that gameplay provides stronger reasoning transfer.”

We trained on a second set of 100 DeepMath problems and a second set of 100 INTELLECT-3-RL code problems. Each set is disjoint from the first. We used the same base model and training settings for 100 gradient steps. At 32K, GlyphBench RL reaches 63.48% on Reasoning Gym. On the new sets, Math RL reaches 60.71% and Code RL reaches 61.62%. Figure 8 compares the strongest baseline in each domain at 32K. The table below gives accuracy (%) for both training sets at each response budget (Appendix I.1).

Policy2K8K16K32K
GlyphBench RL23.4053.3362.5863.48
Math RL (1)30.1656.9162.2062.29
Math RL (2)28.3955.3960.6560.71
Code RL (1)24.9551.7358.6558.98
Code RL (2)31.2255.5861.2061.62

“Clarification: How do the game, math, and code training runs compare in distinct training instances, total generated tokens, optimization steps, and checkpoint-selection criteria? These quantities are needed to interpret the transfer comparison.”

We use a fixed budget of 100 gradient steps for the transfer comparison. At step 100, the game policy has generated 2.534B output tokens over 105,052 rollouts. On the new training sets, Math RL generated 1.207B output tokens over 102,400 rollouts and Code RL generated 2.126B over 102,432 rollouts. The game policy averages 43.4 turns per rollout. Math and code use one turn.

Appendix I.1 gives the training curves and held-out results. Held-out reward is mean return for games and the fraction of correct answers for math and code.

PolicyStepRolloutsOutput tokens (B)Mean turnsHeld-out reward
GlyphBench RL5051,9681.31644.20.271
GlyphBench RL100105,0522.53443.40.319
Math RL (2)5051,2000.6481.00.922
Math RL (2)100102,4001.2071.00.961
Code RL (2)5051,2001.1371.00.383
Code RL (2)100102,4322.1261.00.383

Game RL uses a mixture of 100 tasks with new environment instances. Each math or code run uses 100 fixed problems. All start from Qwen3.5-4B and use the same objective, learning rate of 5 × 10⁻⁶, group size of 32, and batch size of 1,024. Appendix I.1 describes how we select the training problems.

“Clarification: What specific capability coverage or experimental control does GlyphBench add beyond the closest existing suites? Which results demonstrate the importance of those additions beyond increasing the number of tasks?”

GlyphBench lets us train the same LLM on games with different rules using one RL setup. For example: in MiniGrid and MiniHack, the agent has to locate itself, remember what it has seen, and navigate to a goal; in Craftax it also has to gather resources and survive. All use the same spatial text interface, so we can change which games the policy trains on while keeping the model and trainer fixed.

We build on the ASCII observations in Agentick (Creus Castanyer et al., 2026) and extend them to Unicode. Agentick trained PPO agents from scratch and used supervised fine-tuning for its LLMs, but left RL post-training of LLMs to future work. GlyphBench takes that next step: the LLM plays the games, receives rewards for its own actions, and learns from those episodes. We release the training code we used for both individual games and mixtures (§2 and Appendix A.1).

lmgame-Bench studies transfer from training on a single game to other games and planning tasks. We use GlyphBench to study what one LLM learns across 100 tasks with different rules. After 100 updates, its mean return on new game instances rises from 0.012 to 0.319. We then test that same policy against math- and code-trained policies on the same 3,100 reasoning problems (§5.3).

We have made our best effort to address each of your concerns by clarifying what GlyphBench adds beyond existing suites, reporting rollout and token counts, and repeating the math and code baselines on new, disjoint problem sets. We would be happy to clarify any remaining points.

We respectfully ask that you reconsider your rating in light of our rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the author response

Thank you to the reviewers for their feedback. We made the following changes during the discussion.

  • Repeated math and code training on new data (pYft). We trained for 100 gradient steps on a second set of 100 math problems and a second set of 100 code problems. At a 32K response budget, GlyphBench RL reached 63.48% on Reasoning Gym, compared with 60.71% for the new math run and 61.62% for the new code run (§5.3).

  • Added training curves and a breakdown of transfer results (pYft, 2gSR). We report rollout and token counts for games, math, and code over 100 steps. Games and graphs account for most of GlyphBench’s 32K advantage over the original math run. Both math runs score higher than GlyphBench at 2K and 8K (Appendix I.1 and I.3).

  • Released the full anonymized code and documented the environment API (XYTw). The supplement includes the environments, training and evaluation code, tests, and a guide to adding games. Appendix B.1 shows how to use the environments with another trainer.

  • Clarified how GlyphBench builds on existing suites and added two KORGym games (pYft, 2gSR, XYTw). We explain how researchers can vary the training mixture while keeping the model, spatial interface, and trainer fixed. We also adapted LongCat and Tower of Hanoi for turn-by-turn play and evaluated two models on 25 seeds per game. KORGym’s own verifiers accepted all 100 solutions we tested for each game (Appendix A.1 and B.2).

Following our response, reviewer XYTw raised their score from 2 (weak rejection) to 3 (weak acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.