acceptodds
Under review as a conference paper at ICLR 2027

ALERT: Agents for Language-Guided Execution of Revocable Tasks in Real-Time Strategy Games

Abstract

An AI that can win a game while ignoring human instructions is a frightening AI. Real-time strategy benchmarks measure how well agents play, but winning does not show whether an agent followed a human instruction. We introduce ALERT, a benchmark for executing human instructions that can be revised or revoked during full *Command & Conquer: Red Alert 2* matches. The engine continues to advance during model inference, so instructions must remain in force while the agent reacts to a changing game. A reference harness connects language understanding and planning (System 2) to fast execution (System 1), recording active instructions, controller ownership, and actions. We evaluate requested world-state effects separately from match outcomes and use these records to distinguish interpretation, timing, and control failures, rather than relying only on wins and losses. We also use intent and value models to monitor game replays. Testing our early real-time agent, ALERT-Executor, reveals a failure that action logs alone would miss. In seven stored matches, all seven retreat instructions produced engine-accepted movement orders, but controller ownership ended before confirmed group arrival in six. Issuing an accepted retreat order, therefore, did not mean the agent carried out the retreat. ALERT makes such failures visible and provides a way to study instruction following as System 1 and System 2 agents improve.

Primary area:
datasets and benchmarks
Keywords:
LLM, RTS, harness, self-evolve

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NTKSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses the main issues in existing real-time systems - “How does the model respond to human instructions that change in between ?”. The paper claims that current RTS benchmarks and systems pause the state, wait for the inference to be completed and resume it which is not truly “Real Time”. The paper proposes an instruction-following RTS framework ALERT that connects understanding and execution systems and uses the harness around it to stress test the model performance as the human instruction changes midway. The authors use Command & Conquer: Red Alert 2 and a single game map from it to evaluate their framework. Using 49 instructions across 12 sequences they examine multiple aspects of execution, including instruction acceptance, task management, and verification of world-state effects using 24 hour clock paced claude sonnet runs with a 6 model comparison.

Critical strengths
  1. The paper targets an important problem and is well motivated. The tradeoff between fast execution and adapting the strategy with replacing the human’s changed instruction has a lot of real world applications.
  2. The detailed diagnosis in Sec. 6 by separating instruction separation, execution management, world-effect verification tells us where the agent actually fails to respond to the change in the user request. The failure modes with releasing task ownership prematurely are often overlooked in existing benchmarks and the paper addresses that in theirs.
  3. Distinction between clock based and tick based execution is crucial in evaluating RTS systems where the latency can affect the task outcome.
Critical weaknesses
  1. Limited coverage in the experiments. The paper only tests 49 instructions across 12 sequences and on a single map. This becomes difficult to understand if the proposed approach generalizes to all the environments.
  2. Insufficient verification. The “world-effect” covers 23/79 cases with 18 of them passing. Majority of the cases remain unverified which makes the benchmark unreliable.
  3. Lack of baseline comparison. The proposed approach does not compare with standard techniques like direct LLM-to-action execution or fixed task queues and show if ALERT outperforms these traditional methods.
  4. The historical retreat experiments reveal premature ownership release in 6 of 7 accepted instructions. This shows a substantial task management reliability issue. Although the paper discusses possible solutions to the problem, they don’t show explicitly how their solution works. Since reliable task ownership is central to ALERT's contribution, stronger empirical validation is needed.
Questions for authors
  1. Which Red Alert 2 map was used for evaluation, and how does ALERT's performance generalize to maps with different terrain, sizes, and strategic layouts?
  2. Can the authors provide post-repair results on the same retreat scenarios, including premature ownership-release rates and actual retreat completion rates, to demonstrate that the proposed repair is effective?
  3. Why were only 23 of 79 cases checked for world effects? How were these cases selected, and could selection bias influence the reported 18/23 success rate?
  4. How do model inference latency, clock-paced execution, and tick-paced execution affect the reported results?
Reviewer_u7fmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces ALERT, a benchmark designed to evaluate how agents follow human instructions. Specifically, it focuses on instructs that can be revised/revoked over the course of execution, and focuses on real-time strategy games as the environment (Red Alert 2). It also introduces ALERT-executor, a 3-part system: a reference agent that uses language models for interpretation/planning, a ledger to track instructions/orders, and controllers to execute. The paper shows analysis of around 24 matches and analyzes whether the instructions result in the desired outcomes.

Critical strengths

None

Critical weaknesses
  • Writing and presentation. The text is difficult to follow, with frequent shifts in context fragmented trains of thought. The authors mention use of AI to write the manuscript, and this is clear in a negative way, and makes reading the work very difficult.
  • *Many instructions not validated for successful completion The paper's main contribution is the benchmark, aimed at evaluating whether agents follow/execute instructions that may be revised/revoked over the course of gameplay, however, in the 24-match study, only 23/79 instructions are checked for completion. Therefore the benchmark fails are reliably measuring its primary goal of tracking completed instruction following, particularly when those instructions change.
  • Unclear which agents follows instructions better. The 6-model study primarily evaluates whether instructions become tasks, not if they are carried out. Further, cancellation is demonstrated only in a limited manner. Even further, the benchmark does not reliably distinguish agents that actually follow instructions from those that just register the instructions, which piggybacks on the above point as well as failing to accomplish the main goal.
Questions for authors
  • For the 56 unscored cases, were the 56 instructions successfully completed? Can this remaining evaluation gap be filled? Why are the completion checks missing here?
  • Does cancellation extend beyond the single test case? Do different instruction types/phrasing still work?
Reviewer_A3hwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for AI agents that should take human instruction during their execution in real-time and adjust their actions accordingly. The benchmark is based on a real-time strategy (RTS) game called Command & Conquer: Red Alert 2. The authors design 12 timed instruction sequences to simulate human instructions to the agent with a total of 49 instructions. They aim to evaluate the agents not just based on winning or losing, but also based on the instruction following.

Critical strengths

The paper's goal on evaluating AI agents that take human instructions in real-time is important and significant. I am not familiar with the benchmarks in this regard but the paper suggests there hasn't been enough attention to this topic.

The overall design of the benchmark seems reasonable and the use of gates to evaluate the agent's response to each instruction is clever. It will proved a fine-grained diagnosis.

Critical weaknesses

Most importantly, the paper is very poorly written and is hard to follow. I appreciate that the authors are transparent about the manuscript being AI-generated and only reviewed and revised by human authors. Unfortunately, despite my efforts, it was really hard to understand the contributions. I read the text carefully until end of Section 5. I found myself deciphering every sentence from Section 6 onwards, so I gave up fully understanding the text.

In early sections, the text is not clear about the subject that is being evaluated by this benchmark. I wonder if I should see ALERT as a benchmark for models or agents (i.e. the whole software that may use LLMs in part of the executions). The Table 1 includes both agents (Adaptive Command) and game environments (Orak). The paper mentions ALERT-Executer as a baseline instruction-following RTS framework. I am not sure if other agents can be evaluated by ALERT. If so, it evaluates agent-model pairs and it becomes a stronger contribution. If only the underlying model can change, I the value compared to simpler instruction following benchmarks combined with games becomes more limited.

Beyond this, it appears to me that the work for the benchmark is not complete. The main reported results are said to be on earlier version of ALERT (line 257). Also at least 56 of 98 instructions still don't have implemented completion checks (line 264).

The only evaluation I found was that when running the earlier version of ALERT with Sonnet 5, 18 out of 23 instructions were passed. I found this to be too limited.

Questions for authors

Is ALERT a benchmark just for models or can it be used with any agent scaffolding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.