acceptodds
Under review as a conference paper at ICLR 2027

Red Alert Bench: A Diagnostic Benchmark for Multimodal Planning and Execution

Abstract

Multimodal agents in robotics and other embodied settings need to use observations to plan actions, monitor their effects, and achieve a goal. Final success rates alone do not reveal where this process breaks down. We introduce Red Alert Bench, a diagnostic benchmark and reproducible simulation environment for multimodal planning and execution, based on the real-time strategy game Red Alert. Agents gather resources, build, scout, and fight on a map where actions take effect over time, with configurable observation modality (text, image, or both) and map visibility (limited or full). Researchers can evaluate their own agents on 217 human-authored tasks, each with a verifiable goal, and in one-on-one matches where two agents control opposing sides. Evaluating eight LLMs, we find that text-only input outperforms image-primary input by 7–11 percentage points in each of the four models tested for this comparison. We also find that models repeat blocked commands, such as orders they cannot afford, without adjusting their plans. Agent-to-agent matches reveal a perspective-dependent spatial bias: on a symmetric map, the agent starting on the east side wins 68% of all games and 75% of self-play games. Red Alert Bench thus offers a dynamic world in which agents compete against one another, with execution records that link outcomes to the observations and actions behind them. We release the environment, tasks, evaluation tools, and interaction records.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, multimodal agents, diagnostic benchmarks, planning and execution, spatial reasoning, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bbhVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Red Alert Bench, a benchmark and simulation environment for evaluating multimodal agents on long-horizon planning and execution in a real-time strategy game. The benchmark contains 217 human-authored tasks. It supports text, image, and combined observations, limited and full visibility, mid-task handoff, and one-on-one adversarial evaluation. Experiments involve eight LLMs, a non-LLM action model, and a human baseline. The main findings include better task performance with text-only observations than image-primary observations, frequent repetition of commands that cannot take effect, and a substantial positional bias in competitive matches. In addition to final task outcomes, the benchmark records observations, commands, simulator-state changes, and goal progress to support trajectory-level analysis.

Critical strengths
  1. A broad and potentially useful evaluation environment.

    The benchmark covers a relatively diverse set of long-horizon tasks and supports multiple observation modalities, visibility settings, repeated attempts, mid-task handoff, and adversarial matches. The simulator-based success conditions and detailed execution records are useful additions beyond evaluations based only on final model responses.

  2. The paper identifies concrete relevant agent behaviors.

    The analyses reveal several phenomena, including repeated unaffordable or ineffective commands, differences between issuing a valid command and achieving its intended effect, failure to convert stated plans into tool calls, and strong positional effects in competitive evaluation. The inclusion of human and rule-constrained baselines also helps the understanding of some findings.

Critical weaknesses
  1. Limited scientific contribution beyond behavioral instrumentation. If the intended scope of Red Alert Bench is limited to diagnosing observable behavioral failures, then the paper’s scientific contribution appears relatively narrow. Most of the proposed diagnostics are simulator-specific, hand-defined checks of events such as repeated commands, insufficient resources, action-to-effect conversion, and missing tool calls. Although these measurements are useful for inspecting trajectories, the paper does not yet demonstrate that they yield a general diagnostic methodology or new, transferable understanding of why multimodal agents fail. In its current form, the contribution is therefore closer to a well-instrumented evaluation environment than a systematic framework for agent failure diagnosis. The paper should clarify what generalizable scientific insight Red Alert Bench provides beyond its environment and logging infrastructure.

  2. Limited benchmark inspectability and modality control.

    Observation modality is one of the paper’s main experimental factors, yet the paper does not provide representative examples of the actual inputs supplied under the text-only, image-plus-text, and image-primary conditions. This limits the reader’s ability to assess task difficulty, benchmark validity, and whether the three conditions are information-matched. In particular, because the interfaces differ in the information and naming they expose, the reported performance gap cannot be cleanly interpreted as a visual-perception gap.

Questions for authors
  1. What is the intended scope of “diagnosis” in Red Alert Bench? Is the benchmark intended to identify observable behavioral patterns only, or to attribute failures to underlying capabilities such as perception, planning, execution monitoring, and recovery? What generalizable scientific conclusions should readers draw from the proposed diagnostics?
  2. Could the authors include several representative benchmark examples of the actual text-only, image-plus-text, and image-primary inputs, together with the raw image observation, corresponding text observation, underlying simulator state, task instructions, success predicates, and an example trajectory? This would substantially improve the inspectability of the benchmark.
2 comments

“Although these measurements are useful for inspecting trajectories, the paper does not yet demonstrate that they yield a general diagnostic methodology or new, transferable understanding of why multimodal agents fail.”

First, we would like to clarify that this benchmark is not a synthetic benchmark - it is taken from a real-world gaming environment, along with human annotator-written fine-grained scenarios. Existing benchmarks typically focus on specific aspects of agent capabilities or particular task settings. In contrast, Red Alert Bench provides a unified framework for evaluating and diagnosing a broad range of high-level capabilities relevant to embodied intelligence and robotics. It supports both capability-specific evaluation and diagnosis, such as spatial reasoning, active perception, and planning, and integrated evaluation and diagnosis of how these capabilities work together in dynamic, long-horizon tasks. This unified approach enables us to assess not only individual capabilities, but also how their interactions affect overall agent performance and lead to failures.

To further prove the usefulness of our benchmark on real-world tasks, we conduct the following experiment.

To examine the transferability of capabilities learned from RAB, we fine-tuned Qwen3.5-9B using LoRA on RAB-generated data. The model's accuracy on precise visual localization improved from 20% to 93% on held-out RAB tasks. More importantly, the trained model achieved a 2.0 percentage-point improvement on MeasureBench (Lin et al., 2026) instrument-reading tasks (95% CI [+0.4, +3.7]). These results demonstrate that specific visual-spatial capabilities learned from RAB can transfer to external tasks.

Then, to further assess the generalizability, we compared our model rankings with benchmarks across diverse domains (using Kendall's rank correlation). The τ\tau goes from 0.667 to 0.4; the positive correlations provide evidence that RAB captures aspects of agent performance shared across different evaluation settings.

BenchmarkMatched ModelsKendall's τConcordant Pairsp-value
OmniGameArena Solo40.66783.3%0.333
τ³-Banking80.57178.6%0.061
GDPval-AA80.57178.6%0.061
Terminal-Bench 2.180.50075.0%0.109
Buyout Game50.40070.0%0.483

We also found that the visual-spatial limitations identified by RAB appear in embodied reasoning tasks. By analyzing model errors on ERQA (Gemini Robotics Team et al., 2025), we identified similar difficulties in visual information extraction, spatial reference frames, and reasoning about object interactions. When images were replaced with textual scene descriptions, the model correctly answered 14 of 30 previously challenging questions. Furthermore, under comparable direct-answer settings, model rankings on RAB and ERQA were consistent. These findings further support the relevance of RAB's capability-level diagnostics to embodied intelligence and robotics.

We will include the full experimental results, implementation details, and additional analyses in the revised manuscript and supplementary materials.

“yet the paper does not provide representative examples of the actual inputs supplied under the text-only, image-plus-text, and image-primary conditions.”

“Could the authors include several representative benchmark examples of the actual text-only, image-plus-text, and image-primary inputs, together with the raw image observation, corresponding text observation, underlying simulator state, task instructions, success predicates, and an example trajectory?”

Thank you for the suggestion. We have included some examples of benchmarks in the Appendix of the manuscript, as the rebuttal cannot attach figures.

“What is the intended scope of “diagnosis” in Red Alert Bench?”

Embodied agents and robotic agents are an important novel direction, but they lack sufficient data. Our work provides a similar environment to examine the capability of language models in such a scenario. We hope our work can shed light on the behavior of LLMs operating in an agentic harness that requires capabilities such as complex planning, multi-objective commanding, and multimodality. Our additional experiment on transfer learning above provides a very positive signal of the usefulness of our benchmark.

Thanks again for the careful review. We believe these additional experiments address the concerns you raised. We would greatly appreciate it if you could consider raising your score.

Reviewer_eo6uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Red Alert Bench, a reproducible real-time-strategy simulator with 217 human-authored tasks across 12 task families and 119 paired two-game adversarial matches. Agents receive text-only, image-plus-text, or image-primary observations under limited or full map visibility and issue commands through a shared interface. Eight LLMs, a non-LLM controller, and human players are evaluated using directly verifiable task conditions and execution records. The results show that text-only observations outperform image-primary observations, many valid commands have no effect because agents repeat or cannot afford orders, and adversarial outcomes exhibit a strong starting-position bias that is exposed by swapping sides.

Critical strengths
  1. The benchmark is interesting and has some benefits: it helps identify specific failures, such as repeated commands or action looping. And because it has direct success criteria this is super helpful.

  2. The evaluation covers many different settings. The paper tests observation formats and map visibility on all 217 medium-difficulty tasks. It also studies repeated attempts, agent handoff, and adversarial games. It includes human and non-LLM baselines are included.

  3. I also like the starting-position analysis (The east side wins 68% of the games and 75% of the same-model games, even though the map is symmetric.) This is an interesting bias to consider when comparing agents.

Critical weaknesses
  1. The novelty is limited. TextStarCraft II and LLM-PySC2 already evaluate LLM agents in RTS games, while VLMs Play StarCraft II already studies multimodal observations. TowerMind is especially close. It also uses a lightweight synthetic RTS environment and analyzes invalid or inefficient actions. VisGym already studies multimodal-multiturn interactions with multimodal agents and use it as a test bed to diagnosis certain failure patterns. Red Alert Bench adds more tasks, execution records, and adversarial games. However, I am not sure these additions are enough to justify another synthetic RTS benchmark.
  2. The starting-position bias is interesting, but it mainly shows a bias in this simulator. I am not sure what it tells us about multimodal planning more broadly.
  3. All experiments use the same 2D RTS environment and command interface. The paper does not show whether these findings generalize to other environments.
Questions for authors
  1. What does Red Alert Bench measure that prior RTS benchmarks cannot?

  2. Which findings do the authors expect to generalize beyond this simulator? What evidence supports this?

  3. Are the text and image settings matched for information content? Otherwise, the comparison may reflect easier access to state rather than better planning.

2 comments

We thank the reviewers for the careful review and feedback. We respond to the reviewer’s concern below.

“Red Alert Bench adds more tasks, execution records, and adversarial games. However, I am not sure these additions are enough to justify another synthetic RTS benchmark.”

“The paper does not show whether these findings generalize to other environments.”

“Which findings do the authors expect to generalize beyond this simulator? What evidence supports this?”

First, we would like to clarify that this benchmark is not a synthetic benchmark - it is taken from a real-world gaming environment, along with human annotator-written fine-grained scenarios. Existing benchmarks typically focus on specific aspects of agent capabilities or particular task settings. In contrast, Red Alert Bench provides a unified framework for evaluating and diagnosing a broad range of high-level capabilities relevant to embodied intelligence and robotics. It supports both capability-specific evaluation and diagnosis, such as spatial reasoning, active perception, and planning, and integrated evaluation and diagnosis of how these capabilities work together in dynamic, long-horizon tasks. This unified approach enables us to assess not only individual capabilities, but also how their interactions affect overall agent performance and lead to failures.

Secondly, to further prove the usefulness of our benchmark on realworld tasks, we conduct the following experiment.

To examine the transferability of capabilities learned from RAB, we fine-tuned Qwen3.5-9B using LoRA on RAB-generated data. The model's accuracy on precise visual localization improved from 20% to 93% on held-out RAB tasks. More importantly, the trained model achieved a 2.0 percentage-point improvement on MeasureBench (Lin et al., 2026) instrument-reading tasks (95% CI [+0.4, +3.7]). These results demonstrate that specific visual-spatial capabilities learned from RAB can transfer to external tasks.

Then, to further assess the generalizability, we compared our model rankings with benchmarks across diverse domains (using Kendall's rank correlation). The τ\tau goes from 0.667 to 0.4; the positive correlations provide evidence that RAB captures aspects of agent performance shared across different evaluation settings.

BenchmarkMatched ModelsKendall's τConcordant Pairsp-value
OmniGameArena Solo40.66783.3%0.333
τ³-Banking80.57178.6%0.061
GDPval-AA80.57178.6%0.061
Terminal-Bench 2.180.50075.0%0.109
Buyout Game50.40070.0%0.483

We also found that the visual-spatial limitations identified by RAB appear in embodied reasoning tasks. By analyzing model errors on ERQA (Gemini Robotics Team et al., 2025), we identified similar difficulties in visual information extraction, spatial reference frames, and reasoning about object interactions. When images were replaced with textual scene descriptions, the model correctly answered 14 of 30 previously challenging questions. Furthermore, under comparable direct-answer settings, model rankings on RAB and ERQA were consistent. These findings further support the relevance of RAB's capability-level diagnostics to embodied intelligence and robotics.

We will include the full experimental results, implementation details, and additional analyses in the revised manuscript and supplementary materials.

“The starting-position bias is interesting, but it mainly shows a bias in this simulator.”

To show that this is not due to an environment artifact, we show additional experiments to rule out the environment-induced positional differences. We constructed four layouts on an 80×80 map, each reflected horizontally and vertically to generate four starting configurations with identical relative spatial structures. A rule-based scripted agent successfully completed the task from all four starting corners in nearly identical numbers of turns, whereas the model still exhibited statistically significant differences in success rates across starting positions.

The following table compares the scripted agent and an LLM agent (Gemma) on Layout A - D:

Layout A

Starting CornerScripted Agent (Turns to Success)Success Rate
NW95% (1/20)
NE935% (7/20)
SW95% (1/20)
SE840% (8/20)

Gemma corner-effect p = 0.005.

Layout B

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW75% (1/20)
NE715% (3/20)
SW710% (2/20)
SE725% (5/20)

Gemma corner-effect p = 0.382.

Layout C

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW760% (12/20)
NE760% (12/20)
SW75% (1/20)
SE75% (1/20)

Gemma corner-effect p < 0.001.

Layout D

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW75% (1/20)
NE75% (1/20)
SW710% (2/20)
SE75% (1/20)

Gemma corner-effect p = 1.000.

To investigate the cause, we analyzed the models' movement commands. Gemma consistently favored the northwest quadrant (36.0% of targets vs. 25% expected), and its movements were more consistent in absolute map coordinates than in start-relative coordinates (0.146 vs. 0.248). This preference persisted at temperature 0 (36.3%), suggesting that Gemma's search strategy is anchored to absolute coordinates rather than adapting to its starting position.

We further extended the mirrored-corner experiments to three frontier models: GPT-5.4, Claude Opus 4.7, and Claude Sonnet 4.6. Opus exhibited significant differences in success rates across starting positions (p = 0.002), while GPT-5.4 and Sonnet showed no statistically significant corner effects. Nevertheless, all three models displayed varying degrees of preference for movement targets in the northwest quadrant (27.2%–30.6%, compared with the symmetry-consistent expectation of 25%). Together with Gemma's results, these findings suggest that spatial biases are model-dependent and can arise from agents' coordinate-dependent strategies rather than solely from environmental artifacts.

Thanks again for the careful review. We believe these additional experiments address the concerns you raised. We would greatly appreciate it if you could consider raising your score.

Reviewer_fJJCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Many multimodal agent benchmarks rely on final task success as their primary metric, which obscures where and why agents fail. This paper introduces a benchmark built on the real-time strategy game Red Alert to address that gap. The benchmark supports configurable map visibility (full-map vs. partial) and multiple observation modalities.

The authors define several evaluation settings: agent-vs-agent competition, mid-task handovers, and 217 human-authored tasks with programmatically verifiable goals. At each step, the agent receives the interaction history, the current observation, and a game-related prompt, and outputs the next batch of commands. Tasks are organized into families that probe distinct capabilities, each with multiple difficulty levels.

The evaluation covers eight LLMs, one non-LLM baseline (the decision model Jev), and a human baseline. Key findings include: (1) text-only input outperforms image-primary input across four models; (2) in adversarial play, agents show a “perspective-dependent” spatial bias, with east-side agents winning more often on a symmetric map; and (3) agents frequently repeat unfulfilled requests (e.g., ordering unaffordable units) and repeat orders across turns.

Critical strengths
  1. Well-designed adversarial experiments. Having agents swap sides controls for starting position in paired outcomes, which makes the positional bias measurable.
  2. Evaluation beyond final success. Measuring map exploration, action timing, and command usage gives a deeper view of agent behavior than win rate or task completion alone.
  3. Informative non-LLM baseline. Jev’s results show that producing valid actions is not sufficient for success.
  4. Deterministic, LLM-free judging. Verifiable goals avoid the variance and bias of LLM judges.
  5. Useful difficulty sanity checks. The stall and scripted baselines score 0, and Jev matches the LLMs on the 11 shared tasks, which suggests the tasks are neither trivial nor unreasonably hard.
Critical weaknesses
  1. Limited real-world generalization. The benchmark is a simplified game environment, and it is unclear how performance transfers to real-world settings.
  2. Unclear test of partial observability. Full-map and partial-visibility results are very similar, which raises the question of whether the benchmark actually requires reasoning under partial observability.
  3. Missing comparison to existing benchmarks. The paper does not show how this benchmark compares with other RTS or agent benchmarks in difficulty or model performance. Do model rankings agree? If they do, what new information does this benchmark provide?
  4. Incomplete text vs. image comparison. Results are missing for four of the eight models (Opus, Sonnet, GPT-5.4, Kimi), which weakens the claim that text outperforms image-primary input. In addition, image+text results are very close to text-only. This could mean the map is irrelevant to the tasks, or that the text already encodes the spatial information, and the authors discuss this only briefly.
  5. Unsupported “perspective-dependent” claim. The study never manipulates perspective, so it is unclear how the authors conclude the bias is perspective-dependent rather than, for example, a map or environment artifact.
Questions for authors
  1. Why is the gap between text-only and image+text in Table 2 so small? How do the remaining LLMs perform in this comparison?
  2. What causes the east-side bias? Is it related to the environment (e.g., map layout, unit pathing, UI conventions), and if so, how?
  3. What are the real-world implications if model A outperforms model B on a given set of tasks? In other words, what does a ranking on this benchmark predict about performance elsewhere?
5 comments

We thank the reviewers for the careful review and feedback. We respond to the reviewer’s concern below.

“Limited real-world generalization. The benchmark is a simplified game environment, and it is unclear how performance transfers to real-world settings.” “Missing comparison to existing benchmarks. The paper does not show how this benchmark compares with other RTS or agent benchmarks in difficulty or model performance.”

We would like to first clarify that the Red Alert Bench (RAB) is designed to evaluate high-level capabilities relevant to embodied intelligence and robotics, where collecting large-scale real-world interaction data is challenging. Although simple compared to complex real-world scenarios, its environment captures important challenges faced by embodied agents. It enables systematic evaluation and diagnosis of these capabilities, both individually and in combination.

To examine the transferability of capabilities learned from RAB, we fine-tuned Qwen3.5-9B using LoRA on RAB-generated data. The model's accuracy on precise visual localization improved from 20% to 93% on held-out RAB tasks. More importantly, the trained model achieved a 2.0 percentage-point improvement on MeasureBench (Lin et al., 2026) instrument-reading tasks (95% CI [+0.4, +3.7]). These results demonstrate that specific visual-spatial capabilities learned from RAB can transfer to external tasks.

Then, to further assess the generalizability, we compared our model rankings with benchmarks across diverse domains (using Kendall's rank correlation). The τ\tau goes from 0.667 to 0.4; the positive correlations provide evidence that RAB captures aspects of agent performance shared across different evaluation settings.

BenchmarkMatched ModelsKendall's τConcordant Pairsp-value
OmniGameArena Solo40.66783.3%0.333
τ³-Banking80.57178.6%0.061
GDPval-AA80.57178.6%0.061
Terminal-Bench 2.180.50075.0%0.109
Buyout Game50.40070.0%0.483

We also found that the visual-spatial limitations identified by RAB appear in embodied reasoning tasks. By analyzing model errors on ERQA (Gemini Robotics Team et al., 2025), we identified similar difficulties in visual information extraction, spatial reference frames, and reasoning about object interactions. When images were replaced with textual scene descriptions, the model correctly answered 14 of 30 previously challenging questions. Furthermore, under comparable direct-answer settings, model rankings on RAB and ERQA were consistent. These findings further support the relevance of RAB's capability-level diagnostics to embodied intelligence and robotics.

We will include the full experimental results, implementation details, and additional analyses in the revised manuscript and supplementary materials.

“The paper does not show how this benchmark compares with other RTS or agent benchmarks in difficulty or model performance. ”, “Do model rankings agree? If they do, what new information does this benchmark provide?”

To answer this question, we compared our model rankings with benchmarks across diverse domains (using Kendall's rank correlation). The τ\tau goes from 0.667 to 0.4; the positive correlations provide evidence that RAB captures aspects of agent performance shared across different evaluation settings.

BenchmarkMatched ModelsKendall's τConcordant Pairsp-value
OmniGameArena Solo40.66783.3%0.333
τ³-Banking80.57178.6%0.061
GDPval-AA80.57178.6%0.061
Terminal-Bench 2.180.50075.0%0.109
Buyout Game50.40070.0%0.483

“it is unclear how the authors conclude the bias is perspective-dependent rather than, for example, a map or environment artifact” “What causes the east-side bias? Is it related to the environment (e.g., map layout, unit pathing, UI conventions), and if so, how?”

To show that this is not due to an environment artifact, we show additional experiments to rule out the environment-induced positional differences. We constructed four layouts on an 80×80 map, each reflected horizontally and vertically to generate four starting configurations with identical relative spatial structures. A rule-based scripted agent successfully completed the task from all four starting corners in nearly identical numbers of turns, whereas the model still exhibited statistically significant differences in success rates across starting positions.

The following table compares the scripted agent and an LLM agent (Gemma) on Layout A - D:

Layout A

Starting CornerScripted Agent (Turns to Success)Success Rate
NW95% (1/20)
NE935% (7/20)
SW95% (1/20)
SE840% (8/20)

Gemma corner-effect p = 0.005.

Layout B

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW75% (1/20)
NE715% (3/20)
SW710% (2/20)
SE725% (5/20)

Gemma corner-effect p = 0.382.

Layout C

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW760% (12/20)
NE760% (12/20)
SW75% (1/20)
SE75% (1/20)

Gemma corner-effect p < 0.001.

Layout D

Starting CornerScripted Agent (Turns to Success)Gemma Success Rate
NW75% (1/20)
NE75% (1/20)
SW710% (2/20)
SE75% (1/20)

Gemma corner-effect p = 1.000.

To investigate the cause, we analyzed the models' movement commands. Gemma consistently favored the northwest quadrant (36.0% of targets vs. 25% expected), and its movements were more consistent in absolute map coordinates than in start-relative coordinates (0.146 vs. 0.248). This preference persisted at temperature 0 (36.3%), suggesting that Gemma's search strategy is anchored to absolute coordinates rather than adapting to its starting position.

We further extended the mirrored-corner experiments to three frontier models: GPT-5.4, Claude Opus 4.7, and Claude Sonnet 4.6. Opus exhibited significant differences in success rates across starting positions (p = 0.002), while GPT-5.4 and Sonnet showed no statistically significant corner effects. Nevertheless, all three models displayed varying degrees of preference for movement targets in the northwest quadrant (27.2%–30.6%, compared with the symmetry-consistent expectation of 25%). Together with Gemma's results, these findings suggest that spatial biases are model-dependent and can arise from agents' coordinate-dependent strategies rather than solely from environmental artifacts.

Thanks again for the careful review. We believe these additional experiments address the concerns you raised. We would greatly appreciate it if you could consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.