- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces TextCraft, a text-native strategy game for evaluating LLMs in 1v1 and 2v2 matches. The environment is a gridworld version of a Starcraft-like game, where agents manage resources, production and individual units under partial observability, with simultaneous actions and optional team communication. The evaluation includes an 18-model, 900-game tournament, comparisons of thinking and scratchpad settings, and experiments with learned strategy handbooks and team chat. Teacher generated handbooks improve weaker models against unseen LLM opponents without weight updates. Communication did not clearly help and the effect was unpredictable.
Critical strengths
- The environment provides a useful setting for testing long-term planning alongside tactical execution of LLMs on text environments. Models must balance economic investment with immediate military needs, scout under partial observability and adapt their unit composition and positioning to opponents. The text interface retains these interacting decisions while making the game accessible to LLMs, with both competitive and cooperative play. The in-depth analysis of the tournament results provides insight into the relative strengths and behaviour of different models and the effects of thinking and scratchpad settings.
- The teacher-guided learning results are interesting. Here a stronger model distils game experience into strategy handbooks that improve weaker models without weight updates. These improvements were shown to generalise to unseen opponents, and the analysis of the differences between students and opponents provides insight into where the learned advice transfers and where execution remains difficult. A key takeaway is that the handbooks were able to improve weaknesses or pick better strategies but not improve execution against a stronger opponent that was able to adapt to the student’s strategy.
Critical weaknesses
- Using a fixed harness limits the conclusions that can be drawn about strategic capability. Models are now trained with harnesses and are trained to use code to help their evaluation of scenarios and well as trained to manage their own memory. Using either a more flexible harness or existing coding harnesses would allow the models to better demonstrate their strategic capabilities and would make the evaluation more relevant to real-world applications. The authors acknowledge scaffold dependence, but do not test whether a more flexible harness changes these conclusions.
- Only scripted bots are used as a baseline. Simple RL or human baselines would provide a more informative comparison and help place the LLM results in broader context.
Questions for authors
- Are you able to isolate why the stateless teacher works better? What intuition did you have to test stateless vs stateful?
- Is there a comparison with a handbook generated by the same teacher from the rules alone? This would help distinguish the value of learning from game experience from the value of receiving strategic advice from a stronger model.
- Have you tested same tick communication? The current next-tick delivery means the models may not reach consensus before acting which might encourage a more defensive play style.
- Can you clarify the parameter counts for proprietary models in Table 8? It is not clear what the true parameter counts are for these models however some are definitely misplaced (e.g. Luna I understand to have significantly more than 100B effective params).
- Did you consider using each models highest thinking mode with no scratchpad as the baseline for the swiss tournament? The current approach I feel like needlessly confounds the results.
- Did you consider letting the models interact with the environment using a harness and mcp and allowing them to write code or scripts to help control their units? This has been shown to be effective in other LLM game strategies work (https://llmskirmish.com/screeps). This would also allow the model to manage memory across turns.
- How does TextCraft compare with text-based LLM benchmarks such as BattleAgentBench and Alem, particularly in the control given to agents and the strategic or coordination capabilities tested?
- Have you considered giving weaker models access to minimized unsuccessful and successful game traces in context to evaluate their ability to learn from examples?
- Why did you decide to build a new environment rather than adding a text interface to an existing gridworld rts environment such as Magent2 or Neural MMO (V2)?
6 comments
Rebuttal by Authors [2/N]
Here are our answers to your remaining questions, starting with the one about the environment itself.
Why a new environment, and how it compares with BattleAgentBench, Alem, MAgent2 and Neural MMO.
You asked why we did not add a text interface to an existing gridworld such as MAgent2 or Neural MMO, and how TextCraft compares with other text-based benchmarks. Adding a text interface to one of those would have been a reasonable project. We built our own because we wanted to measure a combination that none of them offers.
| One model controls | Economy and production | Setting | Model acts during play | |
|---|---|---|---|---|
| BattleAgentBench | one tank | none | cooperative and competitive stages | every turn |
| Alem | one avatar | crafting and a tech chain, no army | cooperative | every step |
| LLM Skirmish | all units, through a script | harvesting and spawning | 1v1 | no, code is written between rounds |
| MAgent2 | one unit per agent (RL) | none | team battles | no language interface |
| Neural MMO 2.0 | one avatar per agent (RL) | gathering and a market | free-for-all | no language interface |
| TextCraft | every unit and building of a player | harvesting, construction, research, army production | 1v1 and 2v2 with team chat | every tick |
Each of them has a different focus, such as shifting alliances (BattleAgentBench) or open-ended cooperation over very long episodes (Alem). TextCraft's focus is this combination:
- One model makes both the long-term and the unit-level decisions. It runs a whole side, from the economy to every unit. With one unit per model, or a script written between rounds, some of those decisions are not the model's.
- Decisions are made under uncertainty. With fog of war and simultaneous moves, a model has to scout, infer what the opponent is doing, and commit resources before it can be sure.
- The opponent reacts. This is what separates models in our results: a handbook can hand a weak model a good strategy, but not the ability to change it when Opus takes it apart.
- Weak and strong models can both play. The rules are simple enough that all 18 models fit on one scale. Existing text interfaces to StarCraft II either leave unit control to scripts or expose an action space that is hard for language models.
- It is cheap and self-contained. The engine is plain Python on a CPU, so a game costs essentially the model calls (about 0.25 to 7 USD). MAgent2 and Neural MMO have no construction or army production, and Neural MMO has no collision. Adding these takes changes to their core rules, beyond a text interface.
Overall, we see TextCraft as a benchmark that grows with the models. Because the difficulty comes from the opponent, it gets harder as models improve. We expect new behaviours to appear with stronger models. It is also easy to extend: anyone can add maps, units or mechanics to study anything from strategic reasoning to alignment, for example how models play when civilians are on the battlefield. We will open-source the code.
Why does the stateless teacher work better?
Our intuition was that a stateful teacher has a recency bias: it keeps patching the handbook it already has. The stateless teacher designs it from scratch each time and can spot less obvious regularities across the games. The data are consistent with this. Of the stateful teacher's file reads, 88.6% go to the latest generation. Each of its handbooks shares 52% of its words with the previous one, falling to 37% six generations back. The stateless handbooks share 38–39% at every distance.
Highest thinking with no scratchpad as the baseline.
You are right that this is the simplest baseline. Originally, we wanted every model to enter in the best configuration we tested for it. With older model versions, lower reasoning sometimes played better, and a few open models (Kimi, Qwen Large, GLM Flash) do play better with a scratchpad. In the end, the two approaches mostly coincide: 14 of the 18 models use their highest tested thinking level and 15 play without a scratchpad. We will consider your default for future versions of the leaderboard.
Parameter counts in Table 8.
We agree, and this was an oversight: we were not aware of Luna's effective parameters. We will remove the counts and keep only the coarse Small/Medium/Large tags.
Giving weaker models minimized game traces in context.
We have not run this yet. We expect it to help weaker models less than handbooks do: with traces as examples, the student has to distil the lesson itself, in the same call in which it plays. The rules-only result points the same way, since Haiku does well with a specific plan and poorly with advice it has to interpret. Stronger models may learn directly from examples, which would be an interesting comparison.
Thank you again for your useful feedback. The same-tick communication and the rules-only baseline were both very interesting points, and we are glad you raised them.
I acknowledge this response. I leave my score as is but strongly encourage the authors to consider a human (with appropriate interface) or RL baseline as part of this work rather than future work to position the overall model capabilities and utility of the benchmark.
Rebuttal by Authors
Thank you for the time and effort you have invested in reviewing our work, and for your enthusiasm about it. We are glad you found the environment "a useful setting for testing long-term planning alongside tactical execution" and the teacher-guided learning results interesting. Two of your questions turned into experiments: the rules-only handbook and same-tick communication. Both gave results that we will add to the paper. In this comment, we address the two weaknesses and report those two experiments. We answer your remaining questions in the next one.
W1 and your question about harnesses, MCP and code (LLM Skirmish).
You pointed out that models are now trained to work inside harnesses, with code and their own memory, and that a fixed harness limits what we can conclude. The point is well taken, and we find this direction exciting. We see it as a complementary question more than a missing condition. It helps to separate two kinds of harness.
The first is a script-writing agent. In LLM Skirmish, models write a script between rounds and the script plays the match, so the model never acts during a game. That measures how well a coding agent programs an RTS policy. TextCraft measures the model as the decision-maker during the game: it reads a partial observation every tick and issues economic and unit-level orders against an opponent doing the same.
The second is a tool-using agent that plays the game itself through a harness. We consider this a promising avenue for future work, although it is outside the scope of this paper. We expect two aspects to be important: sandboxing, so that an agent cannot read or alter the engine, and giving every agent a comparable tool budget per tick.
In this work, we opted for a scaffold that no model had seen before, because we built it from scratch. This scaffold is the same for all models, so only the model changes. The cost is the one you point out: we rank models within this scaffold.
W2. RL and human baselines.
You suggested simple RL or human baselines to put the LLM results in context. An RL baseline would be a natural extension. This paper focuses on what language models can do without weight updates. However, the engine is self-contained and runs on CPU, so wrapping it for training is straightforward. Humans are a less direct comparison, because the game is designed for language models and its text interface would be hard for a person to use. People could play through a visual interface like the one in our replays, and how well they would do is an open and interesting question. Both baselines are outside the scope of the current work, but they are interesting and a further avenue for future work.
A handbook from the rules alone.
You asked whether the gain comes from game experience or from the advice of a stronger model. This was a good question, so we ran the control. The same teacher, with the same budget and only the rules, wrote a handbook that won 0 of 10 games for Haiku against Sonnet 5 with high thinking. Four handbooks distilled from the full set of training games (the final generation) won 4, 4, 5 and 7 of 10 (pooled 20/40 vs. 0/10, two-sided Fisher exact test, p=0.003). The rules-only handbook hedges on every mechanic. The experience-derived ones commit to choices that were seen to win, such as about 8 workers, an early barracks and a final push with marauders. So the gain seems to come from experience and not only from a stronger model's advice. Our interpretation is that weaker models do better when following a specific plan than with broad advice, which requires knowing when and how to apply it. This would also explain why handbooks let Haiku and Luna beat mid-range models but not a highly adaptive one like Opus.
Same-tick communication.
You asked whether next-tick delivery keeps teammates from reaching agreement before they act. Thank you for this suggestion.
In the paper, each player makes one model call per tick. This call returns both the orders and the message, so a message can only be read on the next tick. Reaching agreement before acting requires splitting the tick in two: each teammate sends one message before either chooses actions, and both act with the full exchange in context. This is a different setting from ours, with two model calls per tick instead of one. However, we found it interesting and explored it out of curiosity.
For Astra–Fable on the King of the Hill map, the chatting team wins 5 of 8 games in this setting, compared with 0 of 8 with next-tick delivery. In the replays we see models revise orders within the tick, for example redirecting soldiers to an ally's threatened extractor after a request for help. This is consistent with what the paper reports: a chat channel does not by itself ensure coordination, and how it is set up matters. The result comes from 8 games on one map, so it is preliminary. We will extend the experiment and include this setting in the paper.