acceptodds
Under review as a conference paper at ICLR 2027

RTSGameBench: Evaluating VLMs For Strategic Reasoning in Game Environments

Abstract

Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., deciding under uncertainty while coordinating with and adapting to other agents. Real-time strategy (RTS) games can be a natural testbed for this capability, demanding long-horizon planning, coordination with allies, and adaptation to opponents under partial observability. But existing RTS benchmarks cover limited agent configurations, lack controlled evaluation across distinct strategic demands, and remain confined to fixed scenarios. To address these gaps, we present RTSGAMEBENCH, built on Beyond All Reason, an RTS game whose larger scale broadens the strategic space beyond prior testbeds. In RTSGameBench, we offer holistic evaluation across diverse full-game match-ups, diagnostic mini-games each emphasizing a focal strategic demand, and extensible coverage via a selfevolving framework that turns free-form queries into new mini-games, improving with accumulated experience. We further introduce RTSGameAgen, a baseline agent harness that combines scalable group-level trajectory planning with short- and long-term memory and outperforms existing agents. Using it, we find that state-of-the-art VLMs struggle where humans succeed, particularly when they must coordinate with allies or contend with multiple opponents.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Strategic Reasoning, Real-Time Strategy Games

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z6TEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RTSGameBench, a benchmark for evaluating the strategic reasoning of VLMs in the real-time strategy (RTS) game Beyond All Reason (BAR). It comprises four types of full-game match-ups played alongside and against built-in AIs, five diagnostic mini-games that each emphasize one focal strategic demand, and a self-evolving framework that turns free-form user queries into new mini-games. The authors also propose an agent harness, RTSGameAgent. No agent exceeds a win rate of 0.20 in the 3v4 and free-for-all settings, whereas humans reach at least 0.70. The relative strengths of the models vary across mini-games, and self-evolution improves generation quality while reducing token usage.

Critical strengths
  1. Brings team and free-for-all play into the RTS evaluation of VLMs. Existing RTS benchmarks for LLM agents are either limited to 1v1 matches or restricted to single-player or scenario-only tests. This paper covers 1v1, 2v2, 3v4, and four-player free-for-all matches within a single environment, and in team settings the evaluated agent must coordinate with built-in AI allies. This is a major highlight of the work.

  2. Both the mini-games and the generation framework are validated more thoroughly than in comparable work. In a blind rating study with five players who were not involved in designing the benchmark, every mini-game scores highest on its intended strategic demand. The generated games are also verified by running them in the game engine, and the evaluation includes an end-to-end single-pass generation baseline, an ablation of the self-evolution mechanisms, and a blind preference study with four raters. These checks provide evidence for what each mini-game tests and for the playability of the generated games, which makes the benchmark more rigorous than most benchmarks that rely solely on the authors' own claims.

Critical weaknesses
  1. The comparisons behind the main conclusions are underspecified and not on equal footing. The main conclusions compare VLMs with humans, VLMs with one another, and RTSGameAgent with prior agents. The paper states only that the human players used the same interface as the agents, without describing how they operated it or how often they could issue commands. The model results are highly sensitive to the protocol. In Figure 12, extending the inference interval in 1v1 Lv.2 from 1 to 2 minutes cuts the win rate from 0.43 to 0, a drop as large as the full spread across the eight models in that setting. Such a protocol must be fair and fully documented, yet the interval was validated only with RTSGameAgent and Gemini-3.7-Flash. The paper also does not report how and on which model the prompts were developed, the decoding parameters, the error handling, or the total number of model calls per game. It thus remains unclear to what extent the gaps behind the abstract's conclusions stem from the protocol.

  2. The claimed diagnostic and extensible properties lack supporting evidence. These are two of the three advantages the paper claims over prior RTS benchmarks. Yet a rough comparison of Tables 3 and 4 shows that the mini-games produce almost the same model ranking as the full games, so it is unclear what they add. Without a no-op baseline, it is also unclear how much of each win rate reflects the targeted demand. For the generated games, the framework itself rates each game's focal demand and is required to rate it highest. Unlike the hand-designed mini-games, these demands are never rated by blind human raters. The 50 games in Section 5.3 are assessed only for generation quality, and only Appendix H.5 evaluates models on generated games, with five games, two models, and five runs per level. The evidence thus shows that the framework produces playable games that match user requests, but not that it extends the evaluation.

Questions for authors
  1. How did the human players operate the shared interface, and could they act only once per minute or every 3 seconds, like the agents? Could you repeat the inference-interval experiment in Figure 12 with other models and at least one baseline agent to check whether the rankings hold?
  2. What information do the mini-games provide beyond the full games? Could you report the win rate of a no-op policy on each mini-game to show that the win rates reflect the models' decisions?
  3. Could you have blind raters, as in Appendix E.1, rate the strategic demands of a subset of the generated games, and evaluate models on more generated games, with more models and more runs?
6 comments
AuthorsOpenReview ↗

Answers to the questions of Reviewer z6TE (1/5)

We thank Reviewer z6TE for the constructive feedback and recognition of our diverse RTS evaluation settings, including team coordination with AI allies. We also appreciate your acknowledgment of our rigorous validation through blind human assessments, in-engine verification, baseline comparisons, ablation studies, and human preference evaluations. We hope our responses adequately address your comments and concerns.


Critical Weakness 1, Question 1

Insufficient details on the human configuration for evaluation. How did the human players operate the shared interface?

→\to For human evaluation, we developed a dedicated GUI (shown in Figures 38 and 39 in the revised manuscript) that provides the same visual and textual observations and action space as VLM agents across both full games and mini-games. Through this GUI, human players could issue building construction, unit production, group assignment, and group-command trajectories, matching the actions available to VLM agents.


Could humans act only once per minute or every 3 seconds, like the agents?

→\to Yes, human players followed the same decision intervals as VLM agents, with 60 seconds of game time for full games, TCP, and FBA, and 3 seconds for MFD, ATS, and AWD. Humans could issue actions only at these scheduled decision points, rather than continuously during gameplay. As with VLM agents, the game paused during decision-making, and no additional commands could be issued between decision points.


Sensitivity of model comparisons to the inference interval. Could you repeat the inference-interval experiment in Figure 12 with other models and at least one baseline agent to check whether the rankings hold?

→\to Good suggestion! As the reviewer suggested, we updated Fig. 12 in the revised manuscript to include inference-interval comparisons across full-game Duel and all five diagnostic mini-games at Lv.2. Specifically, we compare (i) three VLMs (Gemini-3.7-Flash, GLM-5.3-Flash, and GPT-5.6-Terra) using the same RTSGameAgent framework and (ii) three agent frameworks (RTSGameAgent, LLM-PySC2, and SwarmBrain) using the same Gemini-3.7-Flash backbone. The RTSGameAgent configuration with Gemini-3.7-Flash was evaluated over 30 games per interval, while the other configurations were evaluated over 10 games per interval.

As shown in Fig. 12, win rates generally decrease as inference intervals increase, while the main comparative rankings remain largely consistent across most intervals. For example, Gemini-3.7-Flash generally outperforms the other VLMs on MFD and AWD, while RTSGameAgent maintains an advantage over the baseline agents on these mini-games. However, some rankings change across intervals, particularly when performance differences are small. At longer intervals, performance gaps also narrow as win rates approach zero, especially in full-game Duel.

These results suggest that inference intervals affect absolute performance and can change some rankings, but the main comparative trends are not solely determined by the chosen intervals.

AuthorsOpenReview ↗

Answers to the questions of Reviewer z6TE (2/5)

Critical Weakness 1, Question 1

Missing details about the evaluation protocol. The paper does not report how and on which model the prompts were developed, the decoding parameters, the error handling, or the total number of model calls per game. Without these details, it remains unclear to what extent the gaps behind the abstract's conclusions stem from the protocol rather than model or agent capabilities.

→\to Good point! Below, we clarify the prompting procedures, decoding configurations, model-call counts, and error-handling procedures, and added two tables (Tables 16–17) to the revised manuscript to improve reproducibility and transparency.

Prompt. For evaluation, we use identical prompts and prompting procedures across all models within each setting, without model-specific prompt variants. The full-game prompts are provided in Figs. 15–18, as in the original submission. For diagnostic mini-games, we reuse the same prompt structure, modifying only the game descriptions to reflect the scenario-specific objectives and rules detailed in Appendix E.

Decoding configurations and model calls. We do not tune decoding parameters to optimize benchmark performance. Instead, we use the corresponding model/API defaults, which may differ across models. To make these differences explicit, we have added Table 16 (full games) and Table 17 (diagnostic mini-games) to the revised manuscript. These tables report the reasoning effort, temperature, top-p, top-k, and average number of model calls per game for each model and evaluation setting, along with token usage, execution time, and estimated cost. Dashes indicate parameters for which no applicable default is exposed by the API. These additions make the model-specific inference configurations and computational usage directly comparable.

Error handling. We apply a uniform error-handling protocol across all models and agents. Transient API-level failures (e.g., network errors) are retried without a fixed limit. If a model response fails format or content validation (e.g., invalid unit IDs, coordinates, or unit names), we retry up to 10 times. If validation continues to fail, only valid action components are executed; if no valid actions remain, the corresponding decision interval is skipped. These validation, retry, and fallback procedures are applied consistently across all models, without model-specific or situation-dependent exceptions.

AuthorsOpenReview ↗

Answers to the questions of Reviewer z6TE (3/5)

Critical Weakness 2, Question 2

Lack of supporting evidence for the diagnostic value of mini-games. A comparison of Tables 3 and 4 shows that the mini-games produce almost the same model ranking as the full games, so it is unclear what they add. What evidence supports the claimed diagnostic value of the mini-games beyond full-game evaluation?

→\to Although full games and mini-games show similar overall performance trends, we believe that mini-games provide diagnostic value by revealing demand-specific strengths and weaknesses that are less apparent from full-game outcomes alone. For example, at Lv.2, Gemini-3.7-Flash outperforms Grok-4.6 in full-game duels (0.43 vs. 0.00; Table 3), whereas Grok-4.6 achieves higher WRs on TCP (0.83 vs. 0.67) and FBA (0.57 vs. 0.47) in diagnostic mini-games (Table 4). Conversely, Gemini-3.7-Flash outperforms Grok-4.6 on AWD (0.43 vs. 0.10). We conjecture that these differences reflect Grok-4.6's relative strengths in resource management and opponent modeling, emphasized in TCP and FBA, but weaker performance in adversarial planning, emphasized in AWD.

Beyond these win-rate differences, we further assess the diagnostic value of mini-games through trajectory-level analyses of three VLMs (using the same RTSGameAgent) and human gameplay across all five Lv.1 mini-games (Figs. 33–37). Detailed metric definitions and full trajectory analyses for each mini-game are provided in Appendices I.1 and I.3, respectively. Here, we highlight TCP and ATS as representative examples, illustrating demand-specific behavioral differences in resource management and collaboration, respectively.

In TCP (Resource management, Fig. 33 in the revised manuscript), GPT-5.6-Terra exhibits lower resource and build-power utilization than Gemini-3.7-Flash and GLM-5.3-Flash in later stages, accompanied by slower objective progress. Humans sustain higher resource income and economic investment while maintaining high build-power utilization, reaching the objective earlier than the VLMs. In ATS (Collaboration, Fig. 36 in the revised manuscript), we measure coordination using shared battle participation (the proportion of combat units fighting near the same enemy as allied units) and team distance (the average distance between the agent's units and allied units). Gemini-3.7-Flash and humans exhibit higher shared battle participation and lower team distance than the other VLMs, suggesting more coordinated behavior with allied units.

Together, we believe that these results suggest that mini-games complement full-game evaluation by exposing demand-specific performance differences and associated behavioral patterns that are difficult to identify from overall win rates alone.


Experiments of no-op policy on each mini-game Without a no-op baseline, it is also unclear how much of each win rate reflects the targeted demand. Could you report the win rate of a no-op policy on each mini-game to show that the win rates reflect the models' decisions?

→\to Good suggestion! We additionally evaluated a no-op policy on mini-games, which issues no actions while retaining the game's default unit behaviors, over 30 runs per difficulty level. The results are summarized in Table D.

Table D: Win rates of the no-op policy across diagnostic mini-games (30 runs per setting).

Mini-gameLv.1Lv.2Lv.3
TCP0.000.000.00
MFD0.270.000.00
FBA0.000.000.00
ATS0.370.130.00
AWD0.000.000.00

The no-op policy achieves zero wins across all difficulty levels in TCP, FBA, and AWD, suggesting that passive unit behaviors alone are insufficient for success in these tasks. In contrast, it achieves nonzero WRs in MFD (0.27 at Lv.1) and ATS (0.37/0.13 at Lv.1–2). We conjecture that these wins arise from automatic combat behaviors (e.g., attacking nearby enemies and retaliating when attacked), along with the built-in AI ally in ATS. Nevertheless, Gemini-3.7-Flash achieves substantially higher WRs in MFD (0.93/0.60/0.37) and ATS (1.00/0.40/0.13), as shown in Table 4, suggesting that active unit management and allied coordination contribute to successful gameplay beyond passive unit behaviors. Overall, we believe that the no-op results clarify the contribution of default game behaviors to mini-game outcomes and provide a baseline for assessing the benefits of active agent decisions.

Reviewer_VY4zOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RTSGameBench, a benchmark for evaluating vision-language model (VLM) agents in large-scale, multi-agent real-time strategy (RTS) games, built on Beyond All Reason (BAR) rather than the widely used StarCraft II (SC2). The benchmark has three components: (1) Full Games across four matchup structures—1v1 Duel, 2v2 Symmetric Team, 3v4 Asymmetric Team, and 1v1v1v1 Free-for-All—played against built-in AIs at three difficulty levels; (2) Diagnostic Mini-Games, five controlled scenarios each organized around a focal strategic demand from a taxonomy of RTS AI challenges (Resource Management, Spatial & Temporal Reasoning, Opponent Modeling, Collaboration, Adversarial Planning), at three difficulty levels; and (3) a self-evolving game generation framework that converts free-form user queries into new diagnostic mini-games, reusing validated game design documents, rule sets, and rubrics across generation cycles. The authors also propose RTSGameAgent, a baseline VLM harness combining group-level control (dynamic group assignment and condition-dependent command trajectories) with short- and long-term memory. Experiments show that state-of-the-art VLMs struggle where humans succeed—especially in allied coordination and multi-opponent settings—and that the self-evolving generation framework improves game quality while reducing token usage over successive batches.

Critical strengths
  1. By building on BAR rather than SC2, the benchmark expands the evaluation along several axes that matter for strategic reasoning;
  2. The paper identifies a real gap—existing RTS benchmarks for LLM/VLM agents are mostly SC2-based, text-only, 1v1-focused, or fixed-scenario—and argues for three desiderata: holistic, diagnostic, and extensible evaluation;
  3. The authors recruit five RTS-experienced raters, present mini-games in randomized order under neutral labels without revealing the intended focal demand, and show that every mini-game receives its highest mean rating on its focal demand;
  4. Rather than treating each generation independently, the framework reuses validated artifacts and updates stage-wise rubrics through retrospectives. The staged multi-agent pipeline with rollback and abort handling is well-engineered, and the evaluation along playability, behavioral fidelity, token usage, and human preference is appropriate for an open-ended generation task.
Critical weaknesses
  1. The ablation of group-level control does not isolate the contribution of group-level control. This is my primary concern. Section 5.2 states: "Removing group-level control, which also disables trajectory planning and replanning built on group commands, causes the largest drops in WR and ED." This ablation removes a composite mechanism: group assignment, trajectory planning, and replanning are disabled together, and all three depend on group commands. The reported drop therefore cannot be attributed to group-level control as a component. If, for example, WR falls by 0.36 when the composite is removed, the experiment does not tell us how much of that drop comes from (i) group-level control itself, (ii) the loss of trajectory planning, or (iii) the loss of replanning. A finer-grained ablation—e.g., keeping group-level control but disabling trajectory planning, or keeping trajectory planning but restricting replanning—would separate these factors. As written, the ablation supports only the claim that the combination matters, not that each component contributes as the paper's narrative suggests.
  2. The self-evolving generation framework is evaluated over ten batches of five queries (50 generations total), and the human-preference study uses four raters on 20 game pairs. These are reasonable for a first study but small for claims about the robustness of self-evolution; the paper itself notes that standard deviations overlap in several batches and treats the separation between database reuse and rubric updates as "indicative rather than conclusive." Human results are reported over ten runs and the authors omit confidence intervals because they would be too wide to be informative. This is honest, but it leaves the reader unable to judge how solvable the Asymmetric Team and Free-for-All settings actually are, and therefore how much of the model–human gap reflects model limitations versus task difficulty.
Questions for authors
  1. Can you provide a finer-grained ablation that separates group-level control from trajectory planning and from replanning? For example: (a) group-level control with a single fixed command per group and no trajectory DAG; (b) group-level control with trajectory planning but replanning triggers disabled; (c) full RTSGameAgent. This would let readers attribute the reported WR/ED drops to specific components rather than to the composite.
  2. With 50 generations and overlapping standard deviations, how stable are the self-evolution trends? Would additional batches change the conclusion that database reuse mainly reduces cost while rubric updates mainly improve fidelity? Can you report more human runs, or at least a qualitative account of how humans win in Asymmetric Team and Free-for-All, to establish that these settings are solvable and to calibrate how far current VLMs are from human performance?
6 comments
AuthorsOpenReview ↗

Answers to the questions of Reviewer VY4z (1/3)

We thank Reviewer VY4z for the thoughtful feedback and recognition of our benchmark’s expanded scope beyond StarCraft II and its holistic, diagnostic, and extensible design. We also appreciate your acknowledgment of the blinded human validation of mini-games and the well-engineered self-evolving generation framework and comprehensive evaluations of playability, behavioral fidelity, token efficiency, and human preference. We hope our responses adequately address your comments and concerns.


Critical Weakness 1, Question 1

Ablation for each component’s contributions in RTSGameAgent. The ablation of group-level control does not isolate its contribution, since removing it also disables trajectory planning and replanning built on group commands. Can the authors provide a finer-grained ablation that separates these factors? For example: (a) group-level control with a single fixed command per group and no trajectory DAG; (b) group-level control with trajectory planning but replanning triggers disabled; and (c) full RTSGameAgent.

→\to Good point! Following the reviewer's suggestion, we conducted finer-grained ablation experiments to evaluate the contributions of group-level control, trajectory planning, and replanning separately, as shown in Table C. Specifically, (a) uses group-level control with a single fixed command per group, without trajectory planning or replanning; (b) retains group-level control and trajectory planning but disables replanning, as already evaluated in Table 6; and (c) uses the full RTSGameAgent. We additionally evaluated removing group-level control while retaining per-unit trajectory planning and replanning to more directly isolate its contribution.

Table C. Fine-grained ablation study of RTSGameAgent in full-game duels against Lv.2 built-in AI. Variants (a), (b), and (c) progressively introduce trajectory planning (TP) and event-triggered replanning (RP) under group-level control. We additionally evaluate removing group-level control with and without per-unit TP and RP. We report win rate (WR) and six gameplay metrics, including resources gathered (RG), resources spent (RS), army strength (AS), unit quality (UQ), enemies destroyed (ED), and vision coverage (VC). Higher is better for all metrics.

VariantWRRGRSASUQEDVC
(a) Group-Level Control w/ Fixed Commands (w/o TP & RP)0.100.340.380.390.570.330.30
(b) Group-Level Control w/ TP (w/o RP)0.330.400.440.430.570.450.34
(c) RTSGameAgent (Full)0.430.390.440.440.580.490.33
w/o Group-Level Control (w/o TP & RP)0.070.330.380.380.570.300.29
w/o Group-Level Control (w/ Per-unit TP & RP)0.130.350.390.390.560.350.31

Comparing (a), (b), and (c), WR increases from 0.10 to 0.33 with trajectory planning and further to 0.43 with replanning, while ED increases from 0.33 to 0.45 and 0.49, respectively. To further assess group-level control, we compare its removal both without TP and RP (WR 0.10 vs. 0.07) and with per-unit TP and RP retained (WR 0.43 vs. 0.13). The corresponding ED values also decrease from 0.33 to 0.30 and from 0.49 to 0.35, respectively.

These results suggest that group-level control, trajectory planning, and replanning each contribute to performance, with group-level control providing greater benefits when combined with trajectory planning and replanning. We conjecture that group-level control facilitates coordinated unit movements and engagements, while trajectory planning and replanning enable more effective execution and adaptation of group-level strategies.

AuthorsOpenReview ↗

Answers to the questions of Reviewer VY4z (2/3)

Critical Weakness 2, Question 2

Limited evidence for the robustness of self-evolution. The self-evolving generation framework is evaluated over ten batches of five queries (50 generations total), and the human-preference study uses four raters on 20 game pairs. Given the overlapping standard deviations in several batches, how stable are the self-evolution trends?

→\to Despite overlapping standard deviations in some batches, we find that the overall self-evolution trends remain consistent, with the gaps between Agent w/ SE and Agent w/o SE tending to widen as generation experience accumulates. To further examine this trend, we extended our evaluation from 50 to 100 generations (20 batches of five queries each) and observed similar patterns over the extended evaluation.

As shown in Fig. 6 of the revised manuscript, Agent w/ SE maintains near-perfect playability, progressively reduces token usage, and improves behavioral fidelity toward 1.0, whereas Agent w/o SE shows no comparable sustained improvement. Accordingly, the gaps in behavioral fidelity and token efficiency tend to widen over successive batches. Moreover, human preferences gradually shift toward Agent w/ SE in later batches, providing complementary evidence that the improvements in generation quality are also reflected in human evaluations.

We believe that these results support the consistency of self-evolution benefits over successive generations, despite variability across individual batches.


The contributions of database reuse and rubric updates. Would additional self-evolution batches change the conclusion that database reuse mainly reduces cost while rubric updates mainly improve fidelity?

→\to No. To verify this, we extended the self-evolution ablation experiments from 50 to 100 generations (20 batches of five queries each) and found that the original trends in the relative contributions of database reuse and rubric updates persist, as shown in Fig. 14 (Appendix H.3) of the revised manuscript.

Specifically, removing database reuse leads to an increase in token usage, whereas removing rubric updates leads to a decrease in behavioral fidelity. Moreover, the corresponding gaps relative to Agent w/ SE (Full) generally widen as generation experience accumulates.

These findings align with the roles of the two mechanisms. Database reuse skips redundant generation stages by leveraging validated GDDs and rule sets, while rubric updates incorporate lessons from previous validation failures to improve subsequent generations. Although their contributions are not mutually exclusive, the extended results support our original interpretation.

AuthorsOpenReview ↗

Answers to the questions of Reviewer VY4z (3/3)

Critical Weakness 2, Question 2

Uncertainty in the human baseline and model–human gap. Human results are reported over only ten runs, with confidence intervals omitted due to their limited informativeness. This leaves uncertainty about the solvability of the challenging Asymmetric Team and Free-for-All settings and the extent to which the model–human gap reflects model limitations rather than task difficulty. Can you report more human runs to establish their solvability and calibrate how far current VLMs are from human performance?

→\to To assess task solvability and quantify the model–human performance gap, we expanded the human evaluation at Lv.3 and conducted additional trajectory-level analyses at Lv.1 for Asymmetric Team and Free-for-All.

Additional Human Evaluation. We increased the number of human runs from 10 to 30 per setting at Lv.3. Humans achieved WRs of 0.80 (24/30) in Asymmetric Team and 0.70 (21/30) in Free-for-All, with 90% Wilson confidence intervals of [0.657, 0.893] and [0.551, 0.816], respectively. In contrast, all evaluated VLMs achieved 0.00 WR (0/30) in both settings, with a 90% Wilson upper bound of 0.083. The observed model–human WR gaps are 0.80 and 0.70, respectively, and the human confidence interval lower bounds exceed the VLM upper bound in both settings. These results provide evidence that both challenging Lv.3 settings are solvable by humans under the same evaluation interface, while substantial model–human performance gaps persist despite sampling uncertainty.

Human Gameplay Analysis. To further understand how humans succeed and why VLMs struggle, we analyzed gameplay trajectories from 30 runs per model and human at Lv.1 in both settings, corresponding to the evaluation protocol in Table 3. We introduced two additional metrics, third-party attrition (TPA) for Free-for-All and combat trade efficiency (CTE) for Asymmetric Team (Figs. 31–32). Detailed metric definitions and further analyses are provided in Appendix I of the revised manuscript.

In Free-for-All (1v1v1v1), humans maintain higher defensive investment (RS-D) and vision coverage (VC) than VLMs. They also exhibit higher TPA, indicating that a greater proportion of damage to their opponents is inflicted by other opponents. Humans tend to preserve their forces before engaging more actively later in the game. We conjecture that defensive preparation and threat prioritization help humans manage multi-opponent interactions.

In Asymmetric Team (3v4), humans maintain higher CTE than VLMs, whose values remain closer to the break-even level of 0.5, indicating more favorable combat exchanges despite numerical disadvantage. Humans also preserve greater army strength (AS) toward the end of the game. We conjecture that favorable engagement selection and force preservation contribute to human success under numerical disadvantage.


Can you provide a qualitative account of how humans win in these settings to establish their solvability and calibrate how far current VLMs are from human performance?

→\to Good suggestion! To understand how humans succeed in these challenging settings and why VLMs struggle, we further examined selected human and VLM gameplay replays and provide a qualitative comparison of their strategic behaviors below.

In Asymmetric Team (3v4), humans adopted defensive strategies to compensate for their numerical disadvantage, reinforced allied engagements to achieve favorable combat exchanges, and deployed small groups to repeatedly attack vulnerable enemy positions. In contrast, VLMs tended to consolidate their growing armies into a few large groups, struggling to respond to simultaneous engagements across the battlefield. We conjecture that these limitations in large-scale force management contribute to lower combat trade efficiency (CTE) and subsequent army losses.

In Free-for-All (1v1v1v1), humans used scouting information to assess threats from multiple opponents and distributed their forces between defensive positions and planned attacks. In contrast, VLMs often committed their forces to attacking one opponent while leaving their bases vulnerable to others. We believe that VLMs struggle to account for interactions among multiple opponents, often treating multi-agent conflicts as separate one-on-one engagements.

Reviewer_jCw4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces RTSGameBench, built on Beyond All Reason, for evaluating VLMs strategic reasoning. The benchmarks includes full games across different matchups, diagnostic mini-games, and a pipeline for automatic scenario generation. The authors also propose a custom agent harness with long and short term memory managed by the agent. The agent can make different actions: construct buildings, produces units, assigns units into groups and gives commands to these groups. The actions are then executed by the game. The authors evaluate eight models and compare four harnesses. The results show substantial gaps between agents and humans, particularly when agents must coordinate with allies or face multiple opponents.

Critical strengths
  • I agree with the motivation. RTS games provide a rich environment for evaluating agents. Playing well requires managing the economy and production, controlling an army, scouting, and reacting to what the opponent is doing. These demands interact: a game plan must be supported by the economy, and information gathered through scouting should affect subsequent decisions. This makes RTS games an interesting setting for studying strategic reasoning and adaptation. I also agree with the motivation to explore RTS games beyond StarCraft II. Beyond All Reason provides a useful alternative, with substantial scale and opportunities to study coordination between many units and players.

  • I appreciate the variety of evaluation scenarios. I agree that an RTS benchmark should be holistic, and the combination of full games and smaller scenarios is a strong aspect of this work. The different full-game matchups introduce competitive and cooperative demands, while the mini-games make particular capabilities more relevant. This is especially useful when full-game performance is weak and many win rates are close to zero, the mini-games provide much more signal and reveal differences between agents that would otherwise be difficult to see. The scenario-generation framework also provides a way to expand this coverage. Together, these components create a useful foundation for evaluating agents across a variety of situations.

Critical weaknesses
  • Limited takeaways about why agents perform poorly. For me, this is the biggest issue with the benchmark. The authors provide auxiliary metrics and ablations, but it is still hard to tell how the agents really play and what their limitations are. Poor performance in a scenario intended to emphasize collaboration does not, by itself, explain whether the agent failed to coordinate with its ally or simply made poor economic or combat decisions. The mini-games help identify where performance is weak, but the explanations of why remain largely suggestive.

    I would expect a deeper analysis of at least some raw trajectories, connecting observed behavior to the reported results. For example, plots showing investment in the economy, technology, and army over the course of a match could help explain how a disadvantage develops. Expert commentary on selected replays could also identify mistakes that aggregate metrics miss. This would provide clearer takeaways about micro, macro, and strategy, and make the benchmark more useful for understanding what needs to improve.

  • Missing details about thinking budgets and cost. I could not find clear reporting of reasoning-effort settings, token budgets, or the inference cost of playing a match. I consider this a serious omission. Model performance can depend substantially on the amount of reasoning allowed, so these settings are essential information for interpreting the results. Without them, it is difficult to assess whether the reported differences reflect model capabilities, differences in computational budgets, or both.

    This concern is particularly important because the game pauses during inference. Additional thinking has no cost in game time, making it even more important to specify how much computation each agent receives. The authors need to report the reasoning settings and actual inference usage and cost. A comparison across thinking budgets would provide further insight, but documenting the configuration used for the reported results is a basic requirement for reproducibility.

Questions for authors
  • Why does the evaluation pause the game during inference? My biggest question concerns the decision to pause the game whenever the model is thinking. I understand that the authors want to isolate strategic decision quality, but this changes a defining part of the task. In a real-time strategy game, the opponent continues acting while you think. A good decision can become a bad decision if it arrives too late, and deciding how much time to spend thinking is itself part of playing well. Pausing removes this tradeoff. The underlying environment is an RTS game, but the evaluation does not test whether the agent can make decisions while the game continues. How much do the reported performance and model rankings depend on pausing? Could the authors provide a comparison with the game continuing during inference?
10 comments
AuthorsOpenReview ↗

Answers to the questions of Reviewer jCw4 (1/5)

We thank Reviewer jCw4 for the constructive feedback and agreement with our motivation to use RTS games and Beyond All Reason for evaluating strategic reasoning and adaptation. We also appreciate your recognition of the diverse evaluation scenarios, combining competitive and cooperative full-game matchups, diagnostic mini-games, and extensible scenario generation. We hope our responses address your comments and concerns.


Critical Weakness 1

Deeper analysis of agent behavior through raw trajectories It remains unclear how the agents actually play and why they perform poorly. A deeper analysis of raw trajectories, e.g., investment in economy, technology, and army, would help connect observed behavior to reported results and provide clearer takeaways.

→\to Great point! Following the reviewer's suggestion, we extended our aggregate gameplay analysis (Fig. 5) with trajectory-level analyses of full-game Duel, Free-for-All, Asymmetric Team, and all five diagnostic mini-games at Lv.1, comparing three representative VLMs with human gameplay (Figs. 29–37). We tracked gameplay metrics over time, decomposed resource spending into economy (RS-E), defense (RS-D), and army (RS-A) investments, and introduced setting-specific metrics. Detailed metric definitions and trajectory analyses are provided in Appendix I of the revised manuscript, with the corresponding visualizations presented in Figs. 29–37.

Full-game analysis. In Duel (Fig. 29), humans sustain substantially higher defense investment (RS-D) than VLMs. Gemini-3.7-Flash invests more heavily in army production (RS-A) and achieves unit quality (UQ) and enemies destroyed (ED) comparable to humans, whereas GPT-5.6-Terra shows declining relative economic investment (RS-E), accompanied by lower resource gathering (RG) and ED. These patterns suggest difficulties in balancing economic growth, defense, and military development. In Free-for-All (1v1v1v1) (Fig. 32), VLMs emphasize early army investment while maintaining minimal defensive investment. Their relative economic investment and vision coverage (VC) decline over time, while army strength (AS) stagnates or decreases. Humans maintain stronger defenses and higher third-party attrition (TPA), indicating greater benefits from conflicts among opponents. We conjecture that limited defensive preparation and battlefield awareness contribute to difficulties managing multiple opponents. In Asymmetric Team (3v4) (Fig. 31), VLMs struggle to sustain AS, while their combat trade efficiency (CTE) remains closer to the break-even level of 0.5 than humans. These patterns suggest difficulties maintaining favorable combat exchanges and preserving forces under numerical disadvantage.

Diagnostic mini-game analysis. In TCP (Resource Management, Fig. 33), GPT-5.6-Terra exhibits lower resource and build-power utilization in later stages, accompanied by slower objective progress. Humans sustain economic reinvestment and construction capacity utilization, reaching objectives earlier. In MFD (Spatial & Temporal Reasoning, Fig. 34), Gemini-3.7-Flash maintains relatively high protected-building HP despite losing more army strength, whereas humans preserve more forces while better protecting the most vulnerable building. These differences suggest limitations in balancing force preservation and multi-front defense. In FBA (Opponent Modeling, Fig. 35), GLM-5.3-Flash achieves faster target destruction and higher ED despite lower UQ, whereas GPT-5.6-Terra maintains higher VC but progresses more slowly, revealing behavioral differences despite similar WRs. In ATS (Collaboration, Fig. 36), humans and Gemini exhibit higher shared battle participation and closer allied positioning, while Terra shows better coordination metrics than GLM but lower ED, suggesting that coordination and combat execution contribute differently to performance. In AWD (Adversarial Planning, Fig. 37), Gemini's early defensive investment and sustained AS and ED are consistent with its higher WR, while humans sustain greater investment in both defense and army production.

AuthorsOpenReview ↗

Answers to the questions of Reviewer jCw4 (2/5)

Critical Weakness 1

Expert commentary on selected replays. Expert commentary on selected replays would help connect observed behavior to reported results.

→\to Following the reviewer's suggestion, we conducted expert analyses of selected gameplay replays across all four full-game settings and five diagnostic mini-games, comparing the strategic decisions of VLM agents and humans. We summarize the key observations below, with detailed expert replay analyses provided in Appendix I.2 and I.3 of the revised manuscript.

Full-games. In Duel (1v1), Gemini-3.7-Flash prioritized early territorial expansion and high-value unit production before launching a large-scale offensive. GLM-5.3-Flash instead invested in static defenses before attacking later, while GPT-5.6-Terra focused on repeated attacks and unit production rather than economic reinvestment. In contrast, humans sustained economic growth and base defenses while attacking across multiple fronts. In Symmetric Team (2v2), humans reinforced allied attacks, defended vulnerable allied positions, and even constructed turrets near allied bases. Gemini showed some coordination but increasingly relied on allies for economic development while independently selecting attack targets, suggesting limitations in sustained coordination of economic, defensive, and offensive roles. In Asymmetric Team (3v4), humans compensated for numerical disadvantage through defensive play, timely allied reinforcement, and small-scale raids against vulnerable enemy positions. VLMs instead tended to consolidate their armies into a few large groups, struggling to respond to simultaneous engagements, consistent with their lower combat trade efficiency (CTE) and subsequent army losses. In Free-for-All (1v1v1v1), humans monitored multiple opponents and distributed forces according to anticipated threats and planned attacks. VLMs often attacked one opponent while leaving their bases exposed to others, suggesting difficulty accounting for interactions among multiple opponents.

Diagnostic mini-games. In TCP (Resource Management), humans assigned construction tasks based on worker locations and ongoing activities, reducing unnecessary movement and idle time. VLMs struggled to distribute tasks efficiently as worker counts increased, resulting in underutilized construction capacity and delayed objectives. In MFD (Spatial & Temporal Reasoning), humans prioritized vulnerable buildings based on incoming threats and damage while exploiting terrain advantages. VLMs showed less consistent threat-based force repositioning and terrain-aware decisions. In FBA (Opponent Modeling), humans adapted unit compositions and attack strategies based on scouted enemy defenses, whereas VLMs tended to produce similar unit compositions regardless of observed defenses. In ATS (Collaboration), humans supported allied engagements while deploying some forces to flank vulnerable enemy positions. In contrast, GPT-5.6-Terra and GLM-5.3-Flash often attacked independently without adequately considering allied engagements. In AWD (Adversarial Planning), humans constructed mutually supporting defenses and redistributed forces to threatened fronts. Although VLMs also built defenses, they struggled to coordinate defensive positions and respond to simultaneous attacks, allowing individual fronts to be breached.

AuthorsOpenReview ↗

Answers to the questions of Reviewer jCw4 (3/5)

Critical Weakness 1

Distinguishing collaboration failures from other failures. Poor performance in a scenario intended to emphasize collaboration does not, by itself, explain whether the agent failed to coordinate with its ally or simply made poor economic or combat decisions.

→\to To distinguish collaboration failures from combat or economic failures, we conducted trajectory-level analyses of the ATS mini-game and full-game Symmetric Team (2v2), both at Lv.1, comparing three VLMs with human gameplay. We introduced coordination-specific metrics, including shared battle participation (SBP) and team distance in ATS and ally support response (ASR) in Symmetric Team, and compared them with existing combat and economic metrics (Figs. 30 and 36). Detailed metric definitions are provided in Appendix I.1.

In ATS (Fig. 36), SBP measures the proportion of combat units fighting near the same enemy as allied units, while team distance measures their proximity to allied forces. Humans and Gemini-3.7-Flash exhibit higher SBP and lower team distance than the other VLMs, alongside greater preservation of individual and team army strength. Notably, GPT-5.6-Terra achieves better coordination metrics than GLM-5.3-Flash but lower enemies destroyed (ED) and WR, suggesting that coordination and combat execution contribute differently to overall performance.

In Symmetric Team (2v2) (Fig. 30), ASR measures the damage-weighted proportion of attacks on an ally that the agent responds to by attacking the same opponent within 10 seconds. Humans exhibit substantially higher ASR than VLMs, including Gemini, despite Gemini's relatively high vision coverage and army strength. Humans also sustain greater economic and defensive investment, indicating that coordination-related limitations coexist with broader strategic differences.

Furthermore, our expert replay analysis further shows that humans reinforce allied engagements and protect vulnerable allied positions, including constructing defensive turrets near allied bases. In contrast, Gemini sometimes coordinates attacks with allies but increasingly pursues independent targets. We believe these observations help distinguish limitations in allied coordination from combat and economic shortcomings.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive feedback and encouraging comments on the motivation for using RTS games and Beyond All Reason (jCw4, VY4z), diverse evaluation scenarios covering competitive and cooperative settings (jCw4, z6TE), the combination of full games and diagnostic mini-games (jCw4, VY4z), human validation of the mini-games' intended strategic demands (VY4z, z6TE), the well-engineered self-evolving game generation framework (VY4z), and rigorous validation and supporting experiments (VY4z, z6TE).

We have uploaded the first revision of the manuscript, with changes highlighted in blue.

AuthorsOpenReview ↗

Summary of the Discussion

Dear Reviewers, ACs, SACs and PCs,

We thank you and all reviewers for the time and effort devoted to our paper. As the discussion period ends, we briefly summarize the discussion with each reviewer. All changes are highlighted in blue in the revised manuscript.


Reviewer jCw4 (raised the score from weak rejection to weak acceptance after rebuttal)
→\to The reviewer acknowledged that our responses addressed the remaining concerns and noted that the additional experiments "substantially improved the paper."

  • Agent behavior analysis (W1): trajectory-level analyses, expert replay commentary, and key takeaways on strategic reasoning (App. I).
  • Reasoning budgets and cost (W2): reasoning settings, token usage, and inference cost (Tables 16–17), and compared performance across reasoning-effort levels.
  • Paused evaluation (Q1): clarified the scope of the paused setting and added real-time evaluation results.

Reviewer z6TE (raised the score from weak rejection to weak acceptance after rebuttal)
→\to The reviewer acknowledged that our clarifications and additional experiments addressed reviewer's concerns.

  • Evaluation protocol (W1): detailed the human evaluation protocol, extended the inference-interval study to more models and baseline agents (Fig. 12), and reported evaluation configurations (Tables 16–17).
  • Diagnostic and extensible properties (W2): conducted a no-op baseline, blind human ratings of generated games, and evaluations on additional generated games.

Reviewer VY4z (maintained weak acceptance after rebuttal)
→\to The reviewer described our rebuttal as "thorough and highly responsive," noting that we followed the exact ablation design the reviewer suggested.

  • Fine-grained ablation (W1): separated the contributions of group-level control, trajectory planning, and replanning, as requested.
  • Robustness of results (W2): extended the self-evolution experiments to 100 generations (Figs. 6 and 14) and increased human runs in the challenging settings.

We believe the discussion has substantially strengthened the paper, and all reviewers now recommend acceptance. We sincerely appreciate your time and consideration.

Best regards,
Authors of RTSGameBench

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.