- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RTSGameBench, a benchmark for evaluating the strategic reasoning of VLMs in the real-time strategy (RTS) game Beyond All Reason (BAR). It comprises four types of full-game match-ups played alongside and against built-in AIs, five diagnostic mini-games that each emphasize one focal strategic demand, and a self-evolving framework that turns free-form user queries into new mini-games. The authors also propose an agent harness, RTSGameAgent. No agent exceeds a win rate of 0.20 in the 3v4 and free-for-all settings, whereas humans reach at least 0.70. The relative strengths of the models vary across mini-games, and self-evolution improves generation quality while reducing token usage.
Critical strengths
-
Brings team and free-for-all play into the RTS evaluation of VLMs. Existing RTS benchmarks for LLM agents are either limited to 1v1 matches or restricted to single-player or scenario-only tests. This paper covers 1v1, 2v2, 3v4, and four-player free-for-all matches within a single environment, and in team settings the evaluated agent must coordinate with built-in AI allies. This is a major highlight of the work.
-
Both the mini-games and the generation framework are validated more thoroughly than in comparable work. In a blind rating study with five players who were not involved in designing the benchmark, every mini-game scores highest on its intended strategic demand. The generated games are also verified by running them in the game engine, and the evaluation includes an end-to-end single-pass generation baseline, an ablation of the self-evolution mechanisms, and a blind preference study with four raters. These checks provide evidence for what each mini-game tests and for the playability of the generated games, which makes the benchmark more rigorous than most benchmarks that rely solely on the authors' own claims.
Critical weaknesses
-
The comparisons behind the main conclusions are underspecified and not on equal footing. The main conclusions compare VLMs with humans, VLMs with one another, and RTSGameAgent with prior agents. The paper states only that the human players used the same interface as the agents, without describing how they operated it or how often they could issue commands. The model results are highly sensitive to the protocol. In Figure 12, extending the inference interval in 1v1 Lv.2 from 1 to 2 minutes cuts the win rate from 0.43 to 0, a drop as large as the full spread across the eight models in that setting. Such a protocol must be fair and fully documented, yet the interval was validated only with RTSGameAgent and Gemini-3.7-Flash. The paper also does not report how and on which model the prompts were developed, the decoding parameters, the error handling, or the total number of model calls per game. It thus remains unclear to what extent the gaps behind the abstract's conclusions stem from the protocol.
-
The claimed diagnostic and extensible properties lack supporting evidence. These are two of the three advantages the paper claims over prior RTS benchmarks. Yet a rough comparison of Tables 3 and 4 shows that the mini-games produce almost the same model ranking as the full games, so it is unclear what they add. Without a no-op baseline, it is also unclear how much of each win rate reflects the targeted demand. For the generated games, the framework itself rates each game's focal demand and is required to rate it highest. Unlike the hand-designed mini-games, these demands are never rated by blind human raters. The 50 games in Section 5.3 are assessed only for generation quality, and only Appendix H.5 evaluates models on generated games, with five games, two models, and five runs per level. The evidence thus shows that the framework produces playable games that match user requests, but not that it extends the evaluation.
Questions for authors
- How did the human players operate the shared interface, and could they act only once per minute or every 3 seconds, like the agents? Could you repeat the inference-interval experiment in Figure 12 with other models and at least one baseline agent to check whether the rankings hold?
- What information do the mini-games provide beyond the full games? Could you report the win rate of a no-op policy on each mini-game to show that the win rates reflect the models' decisions?
- Could you have blind raters, as in Appendix E.1, rate the strategic demands of a subset of the generated games, and evaluate models on more generated games, with more models and more runs?
6 comments
Answers to the questions of Reviewer z6TE (2/5)
Critical Weakness 1, Question 1
Missing details about the evaluation protocol. The paper does not report how and on which model the prompts were developed, the decoding parameters, the error handling, or the total number of model calls per game. Without these details, it remains unclear to what extent the gaps behind the abstract's conclusions stem from the protocol rather than model or agent capabilities.
Good point! Below, we clarify the prompting procedures, decoding configurations, model-call counts, and error-handling procedures, and added two tables (Tables 16–17) to the revised manuscript to improve reproducibility and transparency.
Prompt. For evaluation, we use identical prompts and prompting procedures across all models within each setting, without model-specific prompt variants. The full-game prompts are provided in Figs. 15–18, as in the original submission. For diagnostic mini-games, we reuse the same prompt structure, modifying only the game descriptions to reflect the scenario-specific objectives and rules detailed in Appendix E.
Decoding configurations and model calls. We do not tune decoding parameters to optimize benchmark performance. Instead, we use the corresponding model/API defaults, which may differ across models. To make these differences explicit, we have added Table 16 (full games) and Table 17 (diagnostic mini-games) to the revised manuscript. These tables report the reasoning effort, temperature, top-p, top-k, and average number of model calls per game for each model and evaluation setting, along with token usage, execution time, and estimated cost. Dashes indicate parameters for which no applicable default is exposed by the API. These additions make the model-specific inference configurations and computational usage directly comparable.
Error handling. We apply a uniform error-handling protocol across all models and agents. Transient API-level failures (e.g., network errors) are retried without a fixed limit. If a model response fails format or content validation (e.g., invalid unit IDs, coordinates, or unit names), we retry up to 10 times. If validation continues to fail, only valid action components are executed; if no valid actions remain, the corresponding decision interval is skipped. These validation, retry, and fallback procedures are applied consistently across all models, without model-specific or situation-dependent exceptions.
Answers to the questions of Reviewer z6TE (3/5)
Critical Weakness 2, Question 2
Lack of supporting evidence for the diagnostic value of mini-games. A comparison of Tables 3 and 4 shows that the mini-games produce almost the same model ranking as the full games, so it is unclear what they add. What evidence supports the claimed diagnostic value of the mini-games beyond full-game evaluation?
Although full games and mini-games show similar overall performance trends, we believe that mini-games provide diagnostic value by revealing demand-specific strengths and weaknesses that are less apparent from full-game outcomes alone. For example, at Lv.2, Gemini-3.7-Flash outperforms Grok-4.6 in full-game duels (0.43 vs. 0.00; Table 3), whereas Grok-4.6 achieves higher WRs on TCP (0.83 vs. 0.67) and FBA (0.57 vs. 0.47) in diagnostic mini-games (Table 4). Conversely, Gemini-3.7-Flash outperforms Grok-4.6 on AWD (0.43 vs. 0.10). We conjecture that these differences reflect Grok-4.6's relative strengths in resource management and opponent modeling, emphasized in TCP and FBA, but weaker performance in adversarial planning, emphasized in AWD.
Beyond these win-rate differences, we further assess the diagnostic value of mini-games through trajectory-level analyses of three VLMs (using the same RTSGameAgent) and human gameplay across all five Lv.1 mini-games (Figs. 33–37). Detailed metric definitions and full trajectory analyses for each mini-game are provided in Appendices I.1 and I.3, respectively. Here, we highlight TCP and ATS as representative examples, illustrating demand-specific behavioral differences in resource management and collaboration, respectively.
In TCP (Resource management, Fig. 33 in the revised manuscript), GPT-5.6-Terra exhibits lower resource and build-power utilization than Gemini-3.7-Flash and GLM-5.3-Flash in later stages, accompanied by slower objective progress. Humans sustain higher resource income and economic investment while maintaining high build-power utilization, reaching the objective earlier than the VLMs. In ATS (Collaboration, Fig. 36 in the revised manuscript), we measure coordination using shared battle participation (the proportion of combat units fighting near the same enemy as allied units) and team distance (the average distance between the agent's units and allied units). Gemini-3.7-Flash and humans exhibit higher shared battle participation and lower team distance than the other VLMs, suggesting more coordinated behavior with allied units.
Together, we believe that these results suggest that mini-games complement full-game evaluation by exposing demand-specific performance differences and associated behavioral patterns that are difficult to identify from overall win rates alone.
Experiments of no-op policy on each mini-game Without a no-op baseline, it is also unclear how much of each win rate reflects the targeted demand. Could you report the win rate of a no-op policy on each mini-game to show that the win rates reflect the models' decisions?
Good suggestion! We additionally evaluated a no-op policy on mini-games, which issues no actions while retaining the game's default unit behaviors, over 30 runs per difficulty level. The results are summarized in Table D.
Table D: Win rates of the no-op policy across diagnostic mini-games (30 runs per setting).
| Mini-game | Lv.1 | Lv.2 | Lv.3 |
|---|---|---|---|
| TCP | 0.00 | 0.00 | 0.00 |
| MFD | 0.27 | 0.00 | 0.00 |
| FBA | 0.00 | 0.00 | 0.00 |
| ATS | 0.37 | 0.13 | 0.00 |
| AWD | 0.00 | 0.00 | 0.00 |
The no-op policy achieves zero wins across all difficulty levels in TCP, FBA, and AWD, suggesting that passive unit behaviors alone are insufficient for success in these tasks. In contrast, it achieves nonzero WRs in MFD (0.27 at Lv.1) and ATS (0.37/0.13 at Lv.1–2). We conjecture that these wins arise from automatic combat behaviors (e.g., attacking nearby enemies and retaliating when attacked), along with the built-in AI ally in ATS. Nevertheless, Gemini-3.7-Flash achieves substantially higher WRs in MFD (0.93/0.60/0.37) and ATS (1.00/0.40/0.13), as shown in Table 4, suggesting that active unit management and allied coordination contribute to successful gameplay beyond passive unit behaviors. Overall, we believe that the no-op results clarify the contribution of default game behaviors to mini-game outcomes and provide a baseline for assessing the benefits of active agent decisions.
Answers to the questions of Reviewer z6TE (1/5)
We thank Reviewer z6TE for the constructive feedback and recognition of our diverse RTS evaluation settings, including team coordination with AI allies. We also appreciate your acknowledgment of our rigorous validation through blind human assessments, in-engine verification, baseline comparisons, ablation studies, and human preference evaluations. We hope our responses adequately address your comments and concerns.
Critical Weakness 1, Question 1
→ For human evaluation, we developed a dedicated GUI (shown in Figures 38 and 39 in the revised manuscript) that provides the same visual and textual observations and action space as VLM agents across both full games and mini-games. Through this GUI, human players could issue building construction, unit production, group assignment, and group-command trajectories, matching the actions available to VLM agents.
→ Yes, human players followed the same decision intervals as VLM agents, with 60 seconds of game time for full games, TCP, and FBA, and 3 seconds for MFD, ATS, and AWD. Humans could issue actions only at these scheduled decision points, rather than continuously during gameplay. As with VLM agents, the game paused during decision-making, and no additional commands could be issued between decision points.
→ Good suggestion! As the reviewer suggested, we updated Fig. 12 in the revised manuscript to include inference-interval comparisons across full-game Duel and all five diagnostic mini-games at Lv.2. Specifically, we compare (i) three VLMs (Gemini-3.7-Flash, GLM-5.3-Flash, and GPT-5.6-Terra) using the same RTSGameAgent framework and (ii) three agent frameworks (RTSGameAgent, LLM-PySC2, and SwarmBrain) using the same Gemini-3.7-Flash backbone. The RTSGameAgent configuration with Gemini-3.7-Flash was evaluated over 30 games per interval, while the other configurations were evaluated over 10 games per interval.
As shown in Fig. 12, win rates generally decrease as inference intervals increase, while the main comparative rankings remain largely consistent across most intervals. For example, Gemini-3.7-Flash generally outperforms the other VLMs on MFD and AWD, while RTSGameAgent maintains an advantage over the baseline agents on these mini-games. However, some rankings change across intervals, particularly when performance differences are small. At longer intervals, performance gaps also narrow as win rates approach zero, especially in full-game Duel.
These results suggest that inference intervals affect absolute performance and can change some rankings, but the main comparative trends are not solely determined by the chosen intervals.