- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies a controller for an AI agent in a real-time pursuit-evasion game (a chaser trying to stop four survivors from fixing machines), where the high-level plan (which of the four survivors to chase) is decided by an LLM and the low-level actions handed off to RL-trained execution policies.
In particular, the two main elements studied are the time taken by the LLM to decide the high-level plan and the world state encoding that is used represent the current state. Under two different models, the paper shows a consistent choice of 4 seconds maximizes the chaser win rate while the proposed "chessboard" representations work best.
Critical strengths
Hierarchical decision-making may be rather important in real-time environments such as this. Having an LLM do a quick high-level plan that can then be delegated to lower-level execution policies is a natural fit to the problem.
The chessboard encodings; these are much more natural ways to represent spatial relationships to LLMs. Especially in JSON format, since LLMs are being increasing improved on code in the past 1-2 years, these formats will be more recognizable than images or custom text formats.
Critical weaknesses
The motivation of the overall hierarchical setup appeals to the general case, but is then undermined by the application to a very specific (non-standard) benchmark being introduced in this paper. Many of the parameters of the benchmark are fixed: one chaser, four survivors, etc. The fact that there is a sweet spot at 4 seconds makes sense, but there is no way to tell how general these results are; the finding that four seconds represents a sweet spot could entirely be due to the specific benchmark, to the choice of models, to the set of specific RL policies that were learned, etc. For a different real-time environment, different models, different RL algorithm this 4 seconds could easily be 1.2 seconds, or e.g. 6 seconds. The results could be more impactful If the domain had been one used by many previous papers, or if the experiments conducted over a set of different domains.. or even if more than one model class had been compared. In its current form, it feels like quite a specific result.
While it is nice to confirm that the chessboard encodings work better, this would be the expected by most LLM practictioners and ICLR attendees. As such, its significance is unclear.
Using an LLM for the high-level planner seems a bit heavy for this particular problem. With 30B parameters (overhead), a different kind of (and much lighter and simpler) controller can likely be implemented in code, or even a simple rule-based system. Do LLMs specifically add anything valuable here? If the goal is for something like this hierarchical setup to be used to solve larger and harder problems, experiments on a more challenging domain would be more convincing.
Questions for authors
-
Page 1: “frozen LLM” - what is a frozen LLM?
-
"“The staleness score— how well a decision still matches the game state when it executes, scored 0–10 by a blind LLM judge (GLM-5.3-Max (GLM et al., 2024)" Why use an LLM judge for this? It seems like there could be many sensible definitions of a staleness score that could staisfy the intent of the metric. Offloading this to an LLM is seems unnecessary and makes the result the opinion of GLM rather than an appropriately defined metric.
-
Why is the RL baseline so bad? The problem does not seem so hard that RL would struggle here.
A frozen LLM means fixed weights: no fine-tuning, no RL, no online update. The model is conditioned only through the prompt and the input state. Differences across conditions therefore come from what the model sees and how often it sees it, not from model capability.
The RL baseline is not a planner-free setup. Its high level is a rule-based behavior tree and its low level is the PPO controller, with no model anywhere. With the same low-level execution, the model-conditioned runs reach 49.6–62.8 against 41.2 for the baseline. The gap is the high-level planner, rule versus LLM.
On what the model adds over a rule planner, Table 2 already holds that comparison: rule high level with RL execution gives 41.2, and swapping only the high level to the model gives 49.6–62.8. The difference is the model's increment.