- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper discusses the main issues in existing real-time systems - “How does the model respond to human instructions that change in between ?”. The paper claims that current RTS benchmarks and systems pause the state, wait for the inference to be completed and resume it which is not truly “Real Time”. The paper proposes an instruction-following RTS framework ALERT that connects understanding and execution systems and uses the harness around it to stress test the model performance as the human instruction changes midway. The authors use Command & Conquer: Red Alert 2 and a single game map from it to evaluate their framework. Using 49 instructions across 12 sequences they examine multiple aspects of execution, including instruction acceptance, task management, and verification of world-state effects using 24 hour clock paced claude sonnet runs with a 6 model comparison.
Critical strengths
- The paper targets an important problem and is well motivated. The tradeoff between fast execution and adapting the strategy with replacing the human’s changed instruction has a lot of real world applications.
- The detailed diagnosis in Sec. 6 by separating instruction separation, execution management, world-effect verification tells us where the agent actually fails to respond to the change in the user request. The failure modes with releasing task ownership prematurely are often overlooked in existing benchmarks and the paper addresses that in theirs.
- Distinction between clock based and tick based execution is crucial in evaluating RTS systems where the latency can affect the task outcome.
Critical weaknesses
- Limited coverage in the experiments. The paper only tests 49 instructions across 12 sequences and on a single map. This becomes difficult to understand if the proposed approach generalizes to all the environments.
- Insufficient verification. The “world-effect” covers 23/79 cases with 18 of them passing. Majority of the cases remain unverified which makes the benchmark unreliable.
- Lack of baseline comparison. The proposed approach does not compare with standard techniques like direct LLM-to-action execution or fixed task queues and show if ALERT outperforms these traditional methods.
- The historical retreat experiments reveal premature ownership release in 6 of 7 accepted instructions. This shows a substantial task management reliability issue. Although the paper discusses possible solutions to the problem, they don’t show explicitly how their solution works. Since reliable task ownership is central to ALERT's contribution, stronger empirical validation is needed.
Questions for authors
- Which Red Alert 2 map was used for evaluation, and how does ALERT's performance generalize to maps with different terrain, sizes, and strategic layouts?
- Can the authors provide post-repair results on the same retreat scenarios, including premature ownership-release rates and actual retreat completion rates, to demonstrate that the proposed repair is effective?
- Why were only 23 of 79 cases checked for world effects? How were these cases selected, and could selection bias influence the reported 18/23 success rate?
- How do model inference latency, clock-paced execution, and tick-paced execution affect the reported results?