acceptodds
Under review as a conference paper at ICLR 2027

TextCraft: A Text-Native Strategy Benchmark and Training Ground for Language Models

Abstract

Most language-model benchmarks are single-player: models are evaluated against fixed tasks that neither adapt as agents improve nor remain discriminative once performance saturates. Multi-agent competition offers a natural alternative, but existing environments are largely inherited from games designed for humans, where perception and interface complexity can obscure the language-model capabilities of interest. We introduce TextCraft, an open real-time strategy environment (tick-based, simultaneous-move) designed natively for language-model agents. Observations, communication, and actions are expressed entirely in text, while agents must simultaneously manage an economy, a technology tree, and military forces under partial observability across hundreds of sequential decisions. TextCraft supports both 1v1 competition and team-based 2v2 play, requiring long-horizon planning, decision-making under uncertainty, adaptation to an opponent, and coordination with a partner. Because difficulty is induced by other improving agents rather than a fixed answer key, TextCraft has no fixed saturation point and can remain discriminative as model capabilities advance. We release the environment, scripted baselines, and complete game trajectories from a range of open and closed models, enabling analysis of how language-model agents strategize, use partial information, adapt to opponents, and coordinate with teammates. TextCraft also serves as a testbed for improvement rather than measurement alone: With learned natural-language skills and no parameter updates, Claude Haiku 4.5 and Luna 5.6, two lower-ranked models on the leaderboard, achieve competitive win rates against stronger models at the final checkpoint under tournament configurations. Together, these results position TextCraft as a shared environment for measuring and improving long-horizon strategic and multi-agent behavior in language models.

Primary area:
datasets and benchmarks
Keywords:
llm, games, multi-agent systems, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6jkFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TextCraft, a text-native strategy game for evaluating LLMs in 1v1 and 2v2 matches. The environment is a gridworld version of a Starcraft-like game, where agents manage resources, production and individual units under partial observability, with simultaneous actions and optional team communication. The evaluation includes an 18-model, 900-game tournament, comparisons of thinking and scratchpad settings, and experiments with learned strategy handbooks and team chat. Teacher generated handbooks improve weaker models against unseen LLM opponents without weight updates. Communication did not clearly help and the effect was unpredictable.

Critical strengths
  • The environment provides a useful setting for testing long-term planning alongside tactical execution of LLMs on text environments. Models must balance economic investment with immediate military needs, scout under partial observability and adapt their unit composition and positioning to opponents. The text interface retains these interacting decisions while making the game accessible to LLMs, with both competitive and cooperative play. The in-depth analysis of the tournament results provides insight into the relative strengths and behaviour of different models and the effects of thinking and scratchpad settings.
  • The teacher-guided learning results are interesting. Here a stronger model distils game experience into strategy handbooks that improve weaker models without weight updates. These improvements were shown to generalise to unseen opponents, and the analysis of the differences between students and opponents provides insight into where the learned advice transfers and where execution remains difficult. A key takeaway is that the handbooks were able to improve weaknesses or pick better strategies but not improve execution against a stronger opponent that was able to adapt to the student’s strategy.
Critical weaknesses
  • Using a fixed harness limits the conclusions that can be drawn about strategic capability. Models are now trained with harnesses and are trained to use code to help their evaluation of scenarios and well as trained to manage their own memory. Using either a more flexible harness or existing coding harnesses would allow the models to better demonstrate their strategic capabilities and would make the evaluation more relevant to real-world applications. The authors acknowledge scaffold dependence, but do not test whether a more flexible harness changes these conclusions.
  • Only scripted bots are used as a baseline. Simple RL or human baselines would provide a more informative comparison and help place the LLM results in broader context.
Questions for authors
  • Are you able to isolate why the stateless teacher works better? What intuition did you have to test stateless vs stateful?
  • Is there a comparison with a handbook generated by the same teacher from the rules alone? This would help distinguish the value of learning from game experience from the value of receiving strategic advice from a stronger model.
  • Have you tested same tick communication? The current next-tick delivery means the models may not reach consensus before acting which might encourage a more defensive play style.
  • Can you clarify the parameter counts for proprietary models in Table 8? It is not clear what the true parameter counts are for these models however some are definitely misplaced (e.g. Luna I understand to have significantly more than 100B effective params).
  • Did you consider using each models highest thinking mode with no scratchpad as the baseline for the swiss tournament? The current approach I feel like needlessly confounds the results.
  • Did you consider letting the models interact with the environment using a harness and mcp and allowing them to write code or scripts to help control their units? This has been shown to be effective in other LLM game strategies work (https://llmskirmish.com/screeps). This would also allow the model to manage memory across turns.
  • How does TextCraft compare with text-based LLM benchmarks such as BattleAgentBench and Alem, particularly in the control given to agents and the strategic or coordination capabilities tested?
  • Have you considered giving weaker models access to minimized unsuccessful and successful game traces in context to evaluate their ability to learn from examples?
  • Why did you decide to build a new environment rather than adding a text interface to an existing gridworld rts environment such as Magent2 or Neural MMO (V2)?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the time and effort you have invested in reviewing our work, and for your enthusiasm about it. We are glad you found the environment "a useful setting for testing long-term planning alongside tactical execution" and the teacher-guided learning results interesting. Two of your questions turned into experiments: the rules-only handbook and same-tick communication. Both gave results that we will add to the paper. In this comment, we address the two weaknesses and report those two experiments. We answer your remaining questions in the next one.

W1 and your question about harnesses, MCP and code (LLM Skirmish).

You pointed out that models are now trained to work inside harnesses, with code and their own memory, and that a fixed harness limits what we can conclude. The point is well taken, and we find this direction exciting. We see it as a complementary question more than a missing condition. It helps to separate two kinds of harness.

The first is a script-writing agent. In LLM Skirmish, models write a script between rounds and the script plays the match, so the model never acts during a game. That measures how well a coding agent programs an RTS policy. TextCraft measures the model as the decision-maker during the game: it reads a partial observation every tick and issues economic and unit-level orders against an opponent doing the same.

The second is a tool-using agent that plays the game itself through a harness. We consider this a promising avenue for future work, although it is outside the scope of this paper. We expect two aspects to be important: sandboxing, so that an agent cannot read or alter the engine, and giving every agent a comparable tool budget per tick.

In this work, we opted for a scaffold that no model had seen before, because we built it from scratch. This scaffold is the same for all models, so only the model changes. The cost is the one you point out: we rank models within this scaffold.

W2. RL and human baselines.

You suggested simple RL or human baselines to put the LLM results in context. An RL baseline would be a natural extension. This paper focuses on what language models can do without weight updates. However, the engine is self-contained and runs on CPU, so wrapping it for training is straightforward. Humans are a less direct comparison, because the game is designed for language models and its text interface would be hard for a person to use. People could play through a visual interface like the one in our replays, and how well they would do is an open and interesting question. Both baselines are outside the scope of the current work, but they are interesting and a further avenue for future work.

A handbook from the rules alone.

You asked whether the gain comes from game experience or from the advice of a stronger model. This was a good question, so we ran the control. The same teacher, with the same budget and only the rules, wrote a handbook that won 0 of 10 games for Haiku against Sonnet 5 with high thinking. Four handbooks distilled from the full set of training games (the final generation) won 4, 4, 5 and 7 of 10 (pooled 20/40 vs. 0/10, two-sided Fisher exact test, p=0.003p = 0.003). The rules-only handbook hedges on every mechanic. The experience-derived ones commit to choices that were seen to win, such as about 8 workers, an early barracks and a final push with marauders. So the gain seems to come from experience and not only from a stronger model's advice. Our interpretation is that weaker models do better when following a specific plan than with broad advice, which requires knowing when and how to apply it. This would also explain why handbooks let Haiku and Luna beat mid-range models but not a highly adaptive one like Opus.

Same-tick communication.

You asked whether next-tick delivery keeps teammates from reaching agreement before they act. Thank you for this suggestion.

In the paper, each player makes one model call per tick. This call returns both the orders and the message, so a message can only be read on the next tick. Reaching agreement before acting requires splitting the tick in two: each teammate sends one message before either chooses actions, and both act with the full exchange in context. This is a different setting from ours, with two model calls per tick instead of one. However, we found it interesting and explored it out of curiosity.

For Astra–Fable on the King of the Hill map, the chatting team wins 5 of 8 games in this setting, compared with 0 of 8 with next-tick delivery. In the replays we see models revise orders within the tick, for example redirecting soldiers to an ally's threatened extractor after a request for help. This is consistent with what the paper reports: a chat channel does not by itself ensure coordination, and how it is set up matters. The result comes from 8 games on one map, so it is preliminary. We will extend the experiment and include this setting in the paper.

AuthorsOpenReview ↗

Rebuttal by Authors [2/N]

Here are our answers to your remaining questions, starting with the one about the environment itself.

Why a new environment, and how it compares with BattleAgentBench, Alem, MAgent2 and Neural MMO.

You asked why we did not add a text interface to an existing gridworld such as MAgent2 or Neural MMO, and how TextCraft compares with other text-based benchmarks. Adding a text interface to one of those would have been a reasonable project. We built our own because we wanted to measure a combination that none of them offers.

One model controlsEconomy and productionSettingModel acts during play
BattleAgentBenchone tanknonecooperative and competitive stagesevery turn
Alemone avatarcrafting and a tech chain, no armycooperativeevery step
LLM Skirmishall units, through a scriptharvesting and spawning1v1no, code is written between rounds
MAgent2one unit per agent (RL)noneteam battlesno language interface
Neural MMO 2.0one avatar per agent (RL)gathering and a marketfree-for-allno language interface
TextCraftevery unit and building of a playerharvesting, construction, research, army production1v1 and 2v2 with team chatevery tick

Each of them has a different focus, such as shifting alliances (BattleAgentBench) or open-ended cooperation over very long episodes (Alem). TextCraft's focus is this combination:

  • One model makes both the long-term and the unit-level decisions. It runs a whole side, from the economy to every unit. With one unit per model, or a script written between rounds, some of those decisions are not the model's.
  • Decisions are made under uncertainty. With fog of war and simultaneous moves, a model has to scout, infer what the opponent is doing, and commit resources before it can be sure.
  • The opponent reacts. This is what separates models in our results: a handbook can hand a weak model a good strategy, but not the ability to change it when Opus takes it apart.
  • Weak and strong models can both play. The rules are simple enough that all 18 models fit on one scale. Existing text interfaces to StarCraft II either leave unit control to scripts or expose an action space that is hard for language models.
  • It is cheap and self-contained. The engine is plain Python on a CPU, so a game costs essentially the model calls (about 0.25 to 7 USD). MAgent2 and Neural MMO have no construction or army production, and Neural MMO has no collision. Adding these takes changes to their core rules, beyond a text interface.

Overall, we see TextCraft as a benchmark that grows with the models. Because the difficulty comes from the opponent, it gets harder as models improve. We expect new behaviours to appear with stronger models. It is also easy to extend: anyone can add maps, units or mechanics to study anything from strategic reasoning to alignment, for example how models play when civilians are on the battlefield. We will open-source the code.

Why does the stateless teacher work better?

Our intuition was that a stateful teacher has a recency bias: it keeps patching the handbook it already has. The stateless teacher designs it from scratch each time and can spot less obvious regularities across the games. The data are consistent with this. Of the stateful teacher's file reads, 88.6% go to the latest generation. Each of its handbooks shares 52% of its words with the previous one, falling to 37% six generations back. The stateless handbooks share 38–39% at every distance.

Highest thinking with no scratchpad as the baseline.

You are right that this is the simplest baseline. Originally, we wanted every model to enter in the best configuration we tested for it. With older model versions, lower reasoning sometimes played better, and a few open models (Kimi, Qwen Large, GLM Flash) do play better with a scratchpad. In the end, the two approaches mostly coincide: 14 of the 18 models use their highest tested thinking level and 15 play without a scratchpad. We will consider your default for future versions of the leaderboard.

Parameter counts in Table 8.

We agree, and this was an oversight: we were not aware of Luna's effective parameters. We will remove the counts and keep only the coarse Small/Medium/Large tags.

Giving weaker models minimized game traces in context.

We have not run this yet. We expect it to help weaker models less than handbooks do: with traces as examples, the student has to distil the lesson itself, in the same call in which it plays. The rules-only result points the same way, since Haiku does well with a specific plan and poorly with advice it has to interpret. Stronger models may learn directly from examples, which would be an interesting comparison.

Thank you again for your useful feedback. The same-tick communication and the rules-only baseline were both very interesting points, and we are glad you raised them.

I acknowledge this response. I leave my score as is but strongly encourage the authors to consider a human (with appropriate interface) or RL baseline as part of this work rather than future work to position the overall model capabilities and utility of the benchmark.

Reviewer_iZ2YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TextCraft is a text-native, simultaneous-move RTS environment combining economic planning, tactical combat, partial observability, and 1v1 or 2v2 competition. The paper evaluates 18 models in a 900-game tournament, studies teacher-generated natural-language handbooks distilled from students’ gameplay against bots, and compares communication-enabled and silent teams with matched rosters. Frozen handbooks improve two students against some held-out LLM opponents, with substantial variation across training curricula. Communication helps some tested rosters and hurts others. Event-based metrics and linked observation–message–action records support descriptive analysis of these behaviors.

Critical strengths
  • Useful research setting: Combines economic planning, combat, partial observability, and team communication, with linked records that make decisions and their consequences inspectable.
  • Meaningful controls: Frozen-handbook evaluations, curriculum comparisons, and matched-roster chat/no-chat games support informative findings about transfer and communication, including cases where assistance is harmful.
Critical weaknesses
  • Limited incremental contribution: Handbook learning and harmful teammate influence have close precedents. The integrated competitive setting is useful, but the findings offer limited additional methodological or explanatory insight.
  • Key interpretations remain unresolved: Curriculum comparisons show that experience matters, but its added value over same-teacher, no-experience guidance and repeatability across training runs remain unclear. Communication-access effects are supported; agreement-driven plan persistence remains observational. The authors acknowledge these limits, which constrain the insights rather than invalidate the outcomes.
Questions for authors
  1. Which existing result most clearly demonstrates a consequential research capability or empirical insight beyond prior handbook-learning and collaborative-agent studies? What does the competitive setting allow us to establish that those precedents do not?
  2. Is there existing evidence comparing experience-derived handbooks with a same-teacher, no-experience handbook under a matched length budget, or from independent repetitions of the same training condition? This would help distinguish the value and reliability of the learning procedure from the performance of particular frozen handbooks.
  3. Is there evidence beyond the linked observational cases that agreement increases persistence in costly plans? How is the quality of those plans assessed independently of the final outcome? I recognize that Appendix F.4 already distinguishes coordination measurements from causal explanations of winning.
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the time and effort you have invested in reviewing our work. We are glad you found TextCraft a "useful research setting", and that you see our results as supporting "informative findings about transfer and communication". Your points led us to run new experiments, which improved both the paper and our own understanding of what the benchmark shows. In this comment, we address the question of contribution (your first weakness and first question). In the next one, we address the two interpretations you found unresolved (your second weakness, with your second and third questions).

W1 / Q1. What does the competitive setting establish beyond prior handbook-learning and collaborative-agent work?

You noted that handbook learning and harmful teammate influence have close precedents, and asked which of our results most clearly shows something those studies do not, and what the competitive setting adds.

The result we would point to first is that a strategy can be written down and handed to a model, while the ability to adapt it cannot. A handbook lets Haiku, which lost all ten of its tournament games against Sonnet, win 55% of its games against Sonnet with high thinking, because mid-range models like Sonnet tend to play with fixed habits. Against Opus, which changes its strategy in response to its opponent, the same handbook never wins a game. The control you suggested (Q2) supports this reading. Against Sonnet with high thinking, Haiku wins 0 of 10 games with a handbook written from the rules alone, whose hedged advice it has to interpret by itself, and 20 of 40 games with the four handbooks that commit to a specific plan (details in our next comment). A benchmark with fixed answers could not show this, because nothing pushes back against the strategy. This is what the competitive setting adds: the opponent's own capability is the test.

Two further results came out of the same setting, and neither was what we expected:

  • What is learned depends on whom you learn against. With the same procedure, Luna wins 63 of 80 evaluation games when trained against one bot and 23 of 80 when trained against another, although it learns to beat both. Each bot teaches skills that may or may not match how LLMs play.
  • Communication can hurt even when messages are truthful and no teammate is adversarial. None of the 33 building-readiness claims we checked for the expert team were false, yet by tick 50 it has produced less military than its silent counterpart in 22 of 24 games. In these games, the team agrees on plans it cannot sustain.

We agree with you about the precedents. Learning natural-language guidance from experience is established (our main inspiration, acknowledged in the paper, is Didolkar et al., 2025), and so is harmful influence between agents. We use both paradigms as probes and do not propose a new learning method. Our contribution list says "we introduce a method", which suggests more than we intended, and we will reword it.

Our main contribution is the environment: a place where these behaviours can be observed and measured in play against an opponent that reacts. We see TextCraft as a way to keep producing observations of this kind about agentic and multi-agent behaviour. Since the difficulty comes from the opponent, the benchmark gets harder as models improve, and we expect new behaviours to show up as stronger models are added. The same environment could also host weight-update training or RL, as Reviewer 6jkF suggests. In this work we kept to what is native to language models: learning through the prompt and context.

AuthorsOpenReview ↗

Rebuttal by Authors [2/N]

In your second weakness you noted that two interpretations remain unresolved: the added value of experience over guidance from the same teacher without experience, and whether agreement drives persistence in costly plans. We address both here.

Q2. Experience-derived versus no-experience handbooks, and repetitions.

You asked for a handbook written by the same teacher without game experience, under a matched budget, and for independent repetitions of the same training condition. This was a very useful suggestion, thank you. We ran both, for Haiku against Sonnet 5 with high thinking (10 games per handbook, same seeds).

  • No-experience control. The same teacher, with the same budget and only the rules, wrote a handbook that won 0 of 10 games. Four handbooks distilled from the full set of training games (the final generation) won 4, 4, 5 and 7 of 10 (pooled 20/40 vs. 0/10, two-sided Fisher exact test, p=0.003p = 0.003).
  • Repetitions. Those four are the original handbook plus three fresh teacher runs on the same games, so this measures how reliable the distillation step is. They perform similarly (Freeman–Halton exact test, p=0.64p = 0.64; 9–10 of 10 against Luna) and share the same core plan.

Reading the handbooks suggests why experience matters. The rules-only handbook gives hedged advice on every mechanic, with conditions the student has to judge for itself (for example, expand "only when rich and secure"). The experience-derived ones commit to choices that were seen to win, with concrete numbers: about 8 workers, an early barracks, a final push with marauders. Our interpretation is that the hard part for a weaker model is recognising which situation it is in, so it does better with a specific plan than with broad advice. This would also explain why handbooks beat mid-range models but not a highly adaptive one like Opus. We will add both results to the appendix.

Q3. Does agreement increase persistence in costly plans, and how is plan quality assessed independently of outcome?

This is a good observation, and we should clarify what we claim. We do not assert that agreement in general increases persistence in costly plans, and we make no mechanistic claim. The statement comes from what we observed in the Astra–Fable games, and we will make this clear in the wording. As for plan quality, the plans the expert team agrees on are plausible and valid within the rules of the game, for example saving for an additional command centre. Whether they are good plans is hard to judge without looking at outcomes, and we do not have an outcome-free measure of plan quality. If you have one in mind, we would be glad to try it.

What we can check without using the final result is whether a plan was carried out and what it cost. Scratchpads are off in 2v2, so a team's plan is what the teammates agree on in chat.

  • Was the plan carried out? An agreed plan to save for an extra command centre appears in 15 of the 24 expert chat games. Across all 24, only two extra command centres were started and only one was completed.
  • What did holding on to it cost? By tick 50, the chatting team has produced less military than its silent counterpart in 22 of 24 games, and holds more unspent resources (230 vs. 147 on average).
  • Were the messages accurate? Yes: none of the 33 building-readiness claims we checked for Astra and Fable were false, compared with 21 of 25 for Haiku in the beginner team.

So the expert team's agreed plans were mostly not executed, but the team still paid their cost. Our evidence on the role of agreement itself remains observational: the chat condition also shows each model its own recent messages, so we cannot separate a teammate's endorsement from self-commitment.


Thank you again for your review. Your feedback has improved the manuscript. We hope that our responses and the additional analyses address your concerns. We are happy to clarify anything further or answer any additional questions.

Thank you for the additional experiments and clarifications. The same-teacher, rules-only comparison helps address a concrete gap identified in my review and strengthens the empirical contribution. I also appreciate the more carefully scoped claims regarding methodological novelty and the communication findings.

My reservations about independent training repetitions and causal interpretation remain. Nevertheless, these additions, together with the platform’s strengths acknowledged in my original review, shift my overall assessment slightly toward acceptance. I am therefore raising my rating from 2 (Weak Reject) to 3 (Weak Accept).

Reviewer_3rnwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TextCraft, an open-source text-native real-time strategy benchmark and experimental platform for large language models (LLMs). Conventional LLM benchmarks easily hit performance saturation and fail to thoroughly evaluate models’ long-horizon planning, partial-information reasoning, adversarial decision-making and multi-agent collaboration. To fill this gap, the authors build a fully text-interactive RTS environment supporting 1v1 and 2v2 matches with fog-of-war mechanics. They conduct large-scale experiments: a 900-game Swiss-round tournament across 18 LLMs, prompt-only skill distillation without model weight updates, and controlled 2v2 team communication tests. Behaviour clustering is also adopted to identify typical agent gameplay patterns. The core results show that test-time reasoning benefits most models while persistent scratchpad memory has limited effects. Weaker LLMs can greatly boost win rates against stronger opponents by using natural-language strategy handbooks distilled from expert game trajectories, but they cannot replicate top models’ precise dynamic in-game execution. Besides, merely enabling a chat channel in 2v2 games does not guarantee effective teamwork. Communication improves performance only if shared information is truthful and plans are actionable; capable agents may even jointly agree on flawed strategies and get worse results.

Critical strengths
  1. The benchmark captures several important strategic capabilities in one compact interface. Long-horizon planning, resource allocation, tactical action, partial observability, and adaptation all affect the same game outcome, making the environment more informative than a fixed single-step benchmark.

  2. The evaluation infrastructure is unusually useful for reproducibility and behavioral analysis. The common scaffold, side-swapped maps, structured actions, scripted opponents, complete trajectories, and replay tools enable analyses beyond a single win-rate number.

  3. The paper demonstrates a practical prompt-only skill distillation method without model fine-tuning. Stronger policy knowledge from expert trajectories can be compiled into natural-language handbooks and injected into prompts, which significantly improves weaker LLMs’ strategic performance against unseen opponents.

  4. It reveals counterintuitive findings on multi-agent communication in collaborative settings. Simply opening a communication channel does not guarantee better teamwork; the benefit of language-based coordination hinges on truthful information and executable plans rather than communication itself.

Critical weaknesses
  1. The environment’s text-only interface imposes a particular communication bottleneck. All state information is compressed into ASCII text, which may disproportionately penalize models weak at parsing dense text rather than weak at strategic reasoning.

  2. The skill-handbook distillation evaluation lacks rigorous out-of-distribution transfer tests. Handbooks are distilled from specific training opponents, and gains are demonstrated only on a narrow set of unseen LLM opponents. The work does not systematically test transfer to radically different opponent strategies or unfamiliar map layouts.

  3. The 2v2 communication experiments cannot disentangle message quality from agent capability. The design compares chat-on versus chat-off conditions, but it does not isolate whether poor teamwork stems from flawed communication content or from the underlying strategic limits of the paired LLMs.

Questions for authors
  1. Can you systematically audit replays to quantify how often high-win agents exploit edge cases or bugs of the game engine, rather than using general strategic reasoning?

  2. To what extent do your core conclusions hold if you remove the token cap and enable unlimited test-time reasoning?

  3. Can you disentangle how much of the observed ranking and performance patterns come from the fixed prompt scaffolding versus the native capability of each LLM?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the time and effort you have invested in reviewing our work. We are glad you found that the benchmark "captures several important strategic capabilities in one compact interface", that the infrastructure is "useful for reproducibility and behavioral analysis", and that the communication results are counterintuitive. Your questions led us to run new experiments and analyses, and the paper is better because of them.

In this comment, we address the three weaknesses you raised. In the next one, we answer your three questions.

W1. Is the text interface a bottleneck?

You raised the concern that putting the whole game state into text may penalize models that are weak at parsing dense text, so that the ranking would reflect reading and not strategy. This is a fair concern, and we ran a simple experiment to investigate.

Nine models, from first place to last, took a 200-question quiz built from 40 real game observations, with answers checked against the engine's recorded state. Some questions are lookups (e.g. "How many soldier units do you currently have?"); others need one step of inference over the map (e.g. "Which of your units can attack enemy X this tick without moving?"). All nine score 96–100% (guessing gets about 27%). Yet they play at very different levels: Astra scores 100% and wins 96 of its 100 tournament games, Gemini 3.1 Pro also scores 100% but wins 45, and Haiku scores 97% and wins 4.

These results suggest that the textual interface does not bottleneck the models' ability to infer the game state that is available to them. It might still be the case that additional modalities benefit certain models, but a comprehensive study is beyond the scope of this work (given that the proposed benchmark targets large language models).

W2. Out-of-distribution transfer of the handbooks.

You noted that the handbooks are distilled from specific training opponents, and asked for transfer tests against different opponent strategies and unfamiliar maps. This is an important point, and we think we did not explain the design clearly enough in the paper. Both kinds of transfer are tested.

  • Training spans different strategies. The student learns against four scripted bots that play very differently: rush, soldier-only defence, marauder-only economy build, and random play.
  • Evaluation is against a different kind of opponent. Handbooks are tested against LLMs never seen in training. Haiku, which lost all ten tournament games against Sonnet, wins 95% against Sonnet without thinking and 55% with thinking. Luna goes from 15% to 55–75% against Sonnet and from 0% to 60% against Sol. Opus is the limit: Haiku never beats it and Luna wins two games, which suggests that a handbook can help execute a strategy but does not necessarily provide the ability to adapt it to different situations.
  • Unseen maps (Appendix C.4). Each handbook is trained against a single bot on one map, then evaluated against eight LLMs on five maps, four of which were never seen during training. Without a handbook, Luna won 17 of 40 tournament games against four of these eight opponents. Trained on the soldier-only Defensive bot, it learns to use marauders and wins 35 of 40 against the same four, and 63 of 80 against all eight. Trained on the marauder-only Economy bot, it masters that bot but wins only 23 of 80. What transfers seems to depend on whether the bot teaches a counter to what mid-range LLMs usually play.

Thank you again for raising this. The next revision will state the held-out opponents and maps more explicitly in the main text.

W3. Message quality versus agent capability in 2v2.

You pointed out that comparing chat-on with chat-off does not tell us whether poor teamwork comes from what is said or from the strategic limits of the models. We agree that outcomes alone cannot separate the two, and your comment led us to look at this more directly.

Two things help here. First, the chatting and silent teams have the same models, settings, maps and vantages, so capability is held fixed and only the channel differs. Second, every message is linked to the sender's observation and subsequent orders, so we can cross-reference it against the game state. The beginner team's messages are often inaccurate: 21 of Haiku's 25 checked building-readiness claims were false, and its ally waits for reinforcements that do not exist. The expert team's messages are accurate (0 of 33 false), but the teammates agree on plans they cannot afford: by tick 50, they have less military capability than their silent opponents in 22 of 24 games.

So the two teams fail for different reasons: for the beginner team the problem is what is said, while the expert team's messages are accurate and the problem lies in the plans the teammates agree on. Since the silent teams consist of the same models, this points to the communication process more than to the models' strategic limits.

AuthorsOpenReview ↗

Rebuttal by Authors [2/N]

Here are our answers to the three questions you asked.

Q1. Do high-win agents exploit engine bugs or edge cases?

You asked whether we can audit the replays to quantify how often high-win agents win by exploiting the engine and not by strategic reasoning. This is a valid question, and the paper itself lists it as an open limitation, so we looked more closely. We had two frontier models, GPT-6 Astra and Claude Fable 5.2, independently review game replays. Besides reading the games, they wrote and ran scripted checks that compare orders and attacks with the recorded game state, so the result does not rest on a model's judgement alone. Both covered the 270 games played by the three top-ranked models, and the scripted checks ran over all 900 tournament games.

  • In their 270 games we found no violation of the rules for resources, hit points, terrain, attack range, movement speed or rate of fire.
  • None of the three top models ever ordered an attack on a target outside its vision (0 of 19,925 attack orders).
  • Neither review found a case of a model relying on unintended engine behaviour.

This fits how the models interact with the game: they act only through JSON orders, cannot see the engine's code, and in the leaderboard games carry no memory from one game to the next. We want to be careful about what this shows. It does not prove that the engine is free of bugs, or that exploitation is impossible. However, it shows that the best tools we have access to could not find exploits. Given that all replays are publicly available, this check can be repeated by anyone.

Q2. Would the conclusions hold without the token cap?

Thank you for this question. We set the cap to keep the experiments within budget, given that they cover over 350,000 model turns. What we can say is where the cap matters.

During play, the cap was rarely reached. With the thinking settings we used, it was reached in 0.35% of tournament turns (712 of 201,622), and in 40 turns for the five top-ranked models combined. So the results are not explained by responses being cut off, and we expect a raised cap for these same settings to change very little. The cap affected which thinking levels took part. Before the tournament, we left out thinking levels that went over the budget in more than 5% of calls. Without a cap, the models affected could play at a higher level, and we expect that they would play better: in our selection matches, the highest tested level was selected for 14 of the 18 models. Ratings are relative, so these models would move up against the others, and some ranks could change.

The leaderboard should therefore be read as a ranking under a common token cap per turn. The other main conclusions come from comparisons made under the same cap: the same student with and without a handbook, and the same teams with and without chat. We do not expect them to change, but we have not tested this. We state the limitation in the paper and will make it more prominent.

Q3. How much of the ranking comes from the scaffold, and how much from the model?

We agree that separating the two is important, and it was one of the main reasons for the way we designed the evaluation.

We did not use existing agent harnesses, which some models may be more familiar with than others. Instead, we built a new, minimal scaffold for this benchmark, which included the rules, a thinking level and an optional scratchpad (but no strategic advice). Since we built this scaffold from scratch, we could ensure that no model had seen it before. The scaffold is also the same for every model, so it is unlikely to explain the differences between them.


Your questions have made the paper's message clearer. The new analyses suggest that models read the game almost equally well (W1) and that a good strategy can be written down and handed to a weaker model (W2). What separates them is how they adapt that strategy against an opponent that reacts. This is one of the things TextCraft is designed to measure: long-term and short-term strategic decisions that have to be revised as the opponent responds.

Thank you again for your review. We hope these answers and the new analyses address your concerns, and we are happy to answer any further questions.

Thank you for the detailed response. In particular, the new text-comprehension experiment helps address my concern about the text-only interface, the clarification of the out-of-distribution experiments provides more evidence that the distilled handbooks can work in new settings, and the additional message-level analysis helps explain why communication sometimes fails in multi-agent teams. Overall, I think the rebuttal has addressed my main concerns. I will therefore raise my rating from 2 to 3.

We are glad our response addressed your remaining concerns. Thank you for the effort you have invested in reviewing our work and engaging with us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.