- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the behaviour of LLM agents in continuously changing environments, focusing on the impact of model completion latency described as the delta_env. The paper introduces a benchmark built on OpenRA, an open-source library for real-time strategy games, with 2 settings: Fixed-Force and Full-Skirmish. The benchmark measures response time, the game time gap between observation and resulting actions (delta_env), decision frequency and game outcomes. The main experiment add artificial delays before model actions are applied and varies game speed (accelerating or decelerating ticks) to study their effect on final metrics. The authors find that structured observations use fewer input tokens, while hierarchical orchestration control achieves better outcomes in their full-game experiments, but at the cost of using more tokens.
Critical strengths
The OpenRA RTS setting is a promising benchmark direction for agents that must both act at low-latency when the environment continues to evolve, as well as long-term planning to solve the game.
Critical weaknesses
- The main paper underspecifies the experimental setup. While high-level descriptions of the tasks and agents are given, the experimental setup omits critical information to better understand the paper. Specifically, very few details are given on the goals/horizons/interfaces and prompts of the OpenRA games. The hierarchical control scaffold is not explained either. This should be developed in appendix.
- The choice of artificially adding wait time before action is questionable. The authors study the impact of the model using more time to take an action by adding an artificial delay. This is a major confound: all the observations on downstream metrics (cf Figure 2) are the consequences of this choice rather than model behaviour. A cleaner setting would vary the reasoning strength of the model so more latency implies more compute.
Questions for authors
-
Can the experimental give more details on the games, victory rules, action space, Flat vs Hierachical orchestrations.
-
How does this benchmark compare to VideoGameBench, Gaia2 (time split of the benchmark), GameWorld-RT, TickingCollabBench, and RTSGameBench? Please clarify its positioning, novelty, and motivation in the introduction or related work.
-
In addition to the artificial wait times, could you report the effect of varying reasoning effort of a fixed model and report the actual latency and game outcomes? This would actually measure whether more time spent on compute improves or hinders downstream performance in such low-latency setup.