acceptodds
Under review as a conference paper at ICLR 2027

What Should LLMs See, and How Often Should They Plan? World-State Encodings for Real-Time Games

Abstract

Real-time game planning with large language models (LLMs) raises two ordered questions: how often should the planner decide, and how should the world state be encoded? We study *training-free high-level planning* in a one-versus-four asymmetric pursuit game, where a frozen LLM selects tactical tasks and targets while a fixed, pretrained RL controller executes frame-level actions. First, an interval sweep with paired output-token budgets reveals an inverted-U trend in chaser win rate under both planners: performance peaks at s and declines at shorter and longer tested intervals. Shorter intervals pair with small output budgets, yielding shallow thinking and suboptimal plans, while longer ones deliver plans to a board that has moved on. Second, at this fixed interval, we compare plain text, topology, graph-text trees, and a visual minimap across seven maps and two planner models, holding the shared decision context and controller fixed. We propose *chessboard*, a local text grid whose layout mirrors map geometry and whose annotations associate entities with their states, and a JSON variant carrying identical information. Both *chessboard* variants improve the chaser win rate over the no-encoding baseline on every evaluated map under both models. The JSON variant leads on all aggregate gameplay metrics, reaching mean win rates of % and %, respectively and percentage points above the corresponding baselines. These findings highlight planning interval and state organization as complementary design choices for real-time LLM agents: neither more frequent replanning nor additional structure or spatial detail alone guarantees better performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM planning in games, world-state encoding for LLM game agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aCpVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies a controller for an AI agent in a real-time pursuit-evasion game (a chaser trying to stop four survivors from fixing machines), where the high-level plan (which of the four survivors to chase) is decided by an LLM and the low-level actions handed off to RL-trained execution policies.

In particular, the two main elements studied are the time taken by the LLM to decide the high-level plan and the world state encoding that is used represent the current state. Under two different models, the paper shows a consistent choice of 4 seconds maximizes the chaser win rate while the proposed "chessboard" representations work best.

Critical strengths

Hierarchical decision-making may be rather important in real-time environments such as this. Having an LLM do a quick high-level plan that can then be delegated to lower-level execution policies is a natural fit to the problem.

The chessboard encodings; these are much more natural ways to represent spatial relationships to LLMs. Especially in JSON format, since LLMs are being increasing improved on code in the past 1-2 years, these formats will be more recognizable than images or custom text formats.

Critical weaknesses

The motivation of the overall hierarchical setup appeals to the general case, but is then undermined by the application to a very specific (non-standard) benchmark being introduced in this paper. Many of the parameters of the benchmark are fixed: one chaser, four survivors, etc. The fact that there is a sweet spot at 4 seconds makes sense, but there is no way to tell how general these results are; the finding that four seconds represents a sweet spot could entirely be due to the specific benchmark, to the choice of models, to the set of specific RL policies that were learned, etc. For a different real-time environment, different models, different RL algorithm this 4 seconds could easily be 1.2 seconds, or e.g. 6 seconds. The results could be more impactful If the domain had been one used by many previous papers, or if the experiments conducted over a set of different domains.. or even if more than one model class had been compared. In its current form, it feels like quite a specific result.

While it is nice to confirm that the chessboard encodings work better, this would be the expected by most LLM practictioners and ICLR attendees. As such, its significance is unclear.

Using an LLM for the high-level planner seems a bit heavy for this particular problem. With 30B parameters (overhead), a different kind of (and much lighter and simpler) controller can likely be implemented in code, or even a simple rule-based system. Do LLMs specifically add anything valuable here? If the goal is for something like this hierarchical setup to be used to solve larger and harder problems, experiments on a more challenging domain would be more convincing.

Questions for authors
  • Page 1: “frozen LLM” - what is a frozen LLM?

  • "“The staleness score— how well a decision still matches the game state when it executes, scored 0–10 by a blind LLM judge (GLM-5.3-Max (GLM et al., 2024)" Why use an LLM judge for this? It seems like there could be many sensible definitions of a staleness score that could staisfy the intent of the metric. Offloading this to an LLM is seems unnecessary and makes the result the opinion of GLM rather than an appropriately defined metric.

  • Why is the RL baseline so bad? The problem does not seem so hard that RL would struggle here.

1 comment

W1&3: Setup is benchmark-specific and 4s may not generalize; W2: chessboard winning is expected; a 30B LLM may be too heavy; Q2: Why an LLM judge?

  • For W1 and W3——benchmark specificity and LLM necessity, please see Common Q5.
  • For W2——whether the chessboard result is expected, please see Common Q4.
  • For Q2——why an LLM judge for staleness, please see Common Q6.

Q: What is a "frozen LLM"; why is the RL baseline so bad?

A frozen LLM means fixed weights: no fine-tuning, no RL, no online update. The model is conditioned only through the prompt and the input state. Differences across conditions therefore come from what the model sees and how often it sees it, not from model capability.

The RL baseline is not a planner-free setup. Its high level is a rule-based behavior tree and its low level is the PPO controller, with no model anywhere. With the same low-level execution, the model-conditioned runs reach 49.6–62.8 against 41.2 for the baseline. The gap is the high-level planner, rule versus LLM.

On what the model adds over a rule planner, Table 2 already holds that comparison: rule high level with RL execution gives 41.2, and swapping only the high level to the model gives 49.6–62.8. The difference is the model's increment.

Reviewer_w8KtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two design choices for real-time LLM planning: planning intervals and world-state encodings. The authors use a hierarchical framework in a one-versus-four pursuit game, where a frozen LLM selects high-level tasks and targets, and a pretrained RL controller executes low-level actions. First, they evaluate different planning intervals and find that performance peaks at 4 seconds. Second, they compare several world-state representations and propose a chessboard encoding and its JSON variant to organize spatial information. Experiments with two Qwen models across seven maps show that the JSON variant achieves the highest win rates, improving over the no-encoding baseline by 7.8 and 7.2 percentage points. The authors also provide ablation studies and LLM-as-judge evaluations to analyze the effects of representation design.

Critical strengths

Extensive Evaluation in a Real-Time Environment. The paper evaluates LLM planning in a closed-loop game environment across seven maps and two planner models, using game outcomes as the main evaluation metric. This provides useful empirical evidence of how different planning configurations perform under real-time constraints.

Critical weaknesses
  1. The paper discusses limitations of existing state representations, but does not convincingly explain why these limitations are a major obstacle for real-time LLM planning. The arguments against frontier models and fine-tuning mainly justify the experimental setting and do not establish a clear research gap. The paper studies planning intervals and world-state encodings, but their connection is not well developed. Although both relate to time-constrained planning, the paper simply calibrates the interval before comparing encodings. The motivation for studying both problems in the same paper remains unclear.

  2. The planning component mainly consists of tuning a fixed interval within an existing hierarchical LLM-RL framework. Finding that 4 seconds works best in this setting is useful, but the methodological contribution is limited. The proposed encoding mainly reorganizes existing state information into a spatial grid or JSON format. Since the JSON variant performs better than the text grid, the contribution of the chessboard layout itself remains unclear. A simpler non-grid JSON baseline would help clarify this.

Questions for authors
  1. What is the main research gap addressed by this paper? Could the authors clarify the connection between planning intervals and world-state encodings, beyond the need to calibrate the interval before comparing representations?
  2. What evidence supports the specific contribution of the chessboard layout? Given that the JSON variant performs better, how can the authors distinguish the benefits of the proposed spatial representation from those of explicit, structured attributes?
  3. How exactly is staleness measured? Table 5 shows nearly constant generation latency across planning intervals, while the staleness score drops substantially. Could the authors explain how the planning interval affects this score and specify the timestamps used for evaluation?
  4. How are generation latency and timeout rate calculated in Table 5? At the 2-second interval, the reported p95 latencies exceed 2 seconds, yet the timeout rates are only 1.2%. Could the authors explain how these statistics are consistent?
1 comment

The four questions map to the common response:

  • For Q1——research gap, please see Common Q3.
  • For Q2——layout versus structured attributes, please see Common Q4.
  • For Q3——staleness measurement and timestamps, please see Common Q6.
  • For Q4——latency and timeout computation, please see Common Q1.
Reviewer_hpKkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a frozen LLM tactical planner paired with a fixed reinforcement-learning controller in a one-versus-four pursuit game, first sweeping the planning interval and then comparing world-state encodings. It reports a four-second optimum in the coupled interval/budget sweep and finds that local chessboard representations, especially a JSON variant, improve aggregate win rates across seven maps and two Qwen planners. The empirical setting is substantial, but internal numerical inconsistencies and unresolved confounding weaken the claimed explanations in terms of shallow reasoning, state staleness, and encoding structure.

Critical strengths
  1. The paper studied an important problem: useful planning under a changing world and a latency budget. A representation that looks clear in a static question may still produce decisions that arrive too late or fail in closed-loop play. Measuring gameplay outcomes under a fixed low-level controller therefore addresses a worthwhile deployment problem, although the evidence is limited to one game and two related planner models.

  2. The paper proposed local chessboard encoding, chessboard JSON serialization, and planning-interval calibration, which are helpful and insightful for the model performance.

  3. The proposed encoding methods achieve strong performance. Table 2 reports chessboard JSON win rates of 62.8 versus 55.0 for the no-encoding baseline with Qwen3-Omni, and 56.8 versus 49.6 with Qwen3.5, gains of 7.8 and 7.2 percentage points. It also beats the strongest reported non-chessboard encoding by 4.6 points on Qwen3-Omni (topology: 58.2) and 5.4 on Qwen3.5 (graph-text tree: 51.4), and leads all seven aggregate gameplay metrics in both model blocks.

Critical weaknesses
  1. There are several numerical statements need reconciliation.

    • Appendix B says 2,000-3,000 frames at 30 frames per second is about 7-10 minutes, but it is about 66.7-100 seconds, or 1.1-1.7 minutes (p. 13, lines 693-694).
    • More seriously, Table 5 cannot satisfy the stated four-survivor win rule if win rate and mean retreats use the same games and weights: when the chaser wins, at most two retreat, so mean retreats cannot exceed 4 minus twice the chaser win probability.
    • The Omni 25-second row reports 43.3 percent wins and 3.32 retreats, exceeding the maximum 3.134; the Omni 20-second row and Qwen3.5 25-second row also violate this bound.
  2. At the two-second interval, Table 5 gives p95 generation latencies of 2.81 and 3.02 seconds but only 1.2 percent timeouts for both models. With the same latency population and a strict two-second deadline, at least about 5 percent must exceed the deadline. Conditional timing or different denominators must be disclosed. Section 4.3 also says every encoding times out below 1 percent at four seconds, whereas the chessboard sweep gives 1.3 percent there, requiring a batch/protocol explanation.

  3. Slimming appears twice in the extracted manuscript, as the graph-text no-slimming ablation and the slimming pass in its discussion, without a sufficiently explicit pruning rule. Please specify which nodes/edges are removed, the thresholds, ordering, and whether information or only serialization length changes.

  4. The interval sweep does not isolate reasoning depth. Planning interval and instructed output budget change together, and only chessboard is used for calibration. Flat generation latency and low timeout rates do not establish that the two-second condition reasons less effectively, nor that four seconds is optimal for other encodings. Feasible controls include varying smaller budgets at a fixed four-second interval and varying the interval with a fixed budget that fits every tested deadline.

Questions for authors

NA

1 comment

For W1——latency and timeout rate, please see Common Q1. For W4——interval versus output budget, please see Common Q2.

W1-1: 2,000–3,000 frames at 30 fps is ~66.7–100 s, not 7–10 minutes.

The count refers to agent-side frames. The agent processes at 5 Hz, five frames per second of observation and decision, while the game client renders and advances the game at 30 fps. The 2,000–3,000 figure is the agent-side count, which gives 7–10 minutes, the range we report; the same span is 12,000–18,000 client frames. The revision will state the agent-side rate and add the conversion.

W1-2: Mean retreats cannot exceed 4 × win rate; the Omni-25 s row (43.3%, 3.32) exceeds 3.134, and the Omni-20 s and Qwen3.5-25 s rows also break the bound.

The retreat column came from a faulty tally. The reviewed values are in the two tables above:

  1. Qwen3-Omni-30B-A3B: | τ (s) | Max out tok. | n | Win (%) | Retreats | p50 (s) | p95 (s) | Timeout (%) | |---|---|---|---|---|---|---|---| | 20 | 775 | 874 | 45.1 | 3.05 | 1.20 | 2.79 | 2.3 | | 25 | 960 | 831 | 43.3 | 3.10 | 1.22 | 2.84 | 2.6 |

  2. Qwen3.5-35B-A3B: | τ (s) | Max out tok. | n | Win (%) | Retreats | p50 (s) | p95 (s) | Timeout (%) | |---|---|---|---|---|---|---|---| | 20 | 640 | 876 | 49.4 | 2.97 | 1.37 | 3.00 | 2.3 | | 25 | 805 | 839 | 49.2 | 2.99 | 1.38 | 3.09 | 2.6 |

W1-3: §4.3 says all encodings stay below 1% at 4 s; the chessboard sweep gives 1.3%.

This will be corrected to "around 1%".

W3: The pruning rule is never specified—which nodes/edges are removed, the thresholds, ordering, and whether only serialization length changes.

Two rules: which entities enter the tree, and how many.

  1. Build the tree: entities are nodes, and the path distance between two entities is the edge label.
  2. First hop: entities within a fixed distance of the chaser.
  3. Second hop: entities one step beyond the first hop.
  4. Cap: each level keeps a fixed number of nodes. When candidates exceed the cap, they are sorted by distance to the chaser and the nearest are kept; the rest, and the edges that touch only them, are not written.

On the four points. What is removed: nodes left outside the per-level cap after sorting, plus the edges that touch only them. Threshold: a per-level node count, with the farthest dropped once it is exceeded. Order: collect both hops first, then sort within each level by distance and keep the first n. Whether only length changes: what changes is how many nodes enter the tree, that is, serialization length and view width, while the retained entities keep their position, state and distance.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common response (1/2)

Q1: At 2 s, Table 5 gives p95 latencies of 2.81 and 3.02 s but only 1.2% timeouts; with the same latency population and a 2 s deadline, at least ~5% should time out. Different denominators must be disclosed.

Two clocks are involved here. The request interval is the sweep setting, 2–25 s, and it fixes how often a decision is asked for. The timeout is a separate, fixed parameter: a call past 4 s is abandoned and the controller falls back to the rule policy, continuing the previous task and target. The timeout column counts the latter.

Generation time is short everywhere and barely moves: p50 is 1.2–1.4 s and p95 falls between 2.75 and 3.09 s. At 2 s, p95 is 2.81 s, about 1.2 s below the ceiling, and 1.2% of calls cross it. A response that arrives after 2 s is still executed as long as it beats the next response; only calls past 4 s are dropped.

The 25 s row differs for another reason. The output cap rises from 60 to 960 tokens, so more responses run long and the share crossing 4 s reaches 2.6%. That follows the cap, not the interval.

The sweep does not isolate reasoning depth: interval and output budget change together, and only chessboard is calibrated. Flat latency and low timeouts do not show that 2 s reasons less, nor that 4 s is optimal elsewhere.

Interval and output cap are set together: the longer the interval, the more tokens are allowed, and the cap is chosen to match the interval (Figure 4, Table 5).

Moving both at once does not prevent telling them apart, because they push the curve in opposite directions. If only the budget mattered, longer intervals would allow more tokens and the win rate would climb monotonically. If only staleness mattered, it would fall monotonically. What we see is a rise and then a fall, peaking at 4 s. Neither factor alone produces that shape.

Each end also has its own measurement:

  • Short interval end: the 2 s level allows 60 tokens, the tightest in the sweep, and its win rate sits 5.5 and 2.2 points below the peak for the two models (Table 5). Its staleness score is the best in the sweep (9.6 and 9.4, Figure 4), so staleness is not what hurts here; the budget is.
  • Long interval end: timeout stays at 1.1–2.6% and generation time hardly changes (p95 2.75–2.84 s and 2.97–3.09 s for the two models, Table 5), which rules out slow responses. What changes is the staleness score, from 9.6 down to 1.6 (9.4 to 1.3 for the second model, Figure 4): the plan was made for a board that has moved on.

The 4 s point was calibrated on the chessboard encoding, so it is not a strict optimum for every encoding. The interval effect comes from two quantities that do not depend on encoding: how fast the board drifts, and how much time a response gets. An encoding changes what the model sees; it does not change how fast the game runs or how long a call takes. A window of 3–5 s should therefore carry over to other encodings in the same environment.

Q3: The connection between the two lines is underdeveloped—the interval is calibrated, then encodings compared—and why they belong in one paper is unclear.

The two questions are not two papers stapled together. They are two layers of one problem: using an LLM to plan inside a real-time game.

The first layer is a precondition. A decision has to be produced at the pace the game runs and be executed before the board changes. The interval belongs here: how often to ask, how much time a call gets, whether the answer still arrives in time. When this layer fails, encoding differences are swamped by execution cost; once it holds, the comparison means something. The effect is not small either: going from 4 s to 25 s costs 17.2 points (Figure 4, Table 5).

The second layer builds on it. Within the same time budget, the question is how to present the board so the model can read it. This is where encoding enters. At a fixed 4 s interval, encodings differ by 7.8 points (Table 2; chessboard JSON 62.8 against 55.0 for no encoding). Both layers move the result by roughly 10 points, so neither is optional, and the encoding comparison is run at an interval that already works.

AuthorsOpenReview ↗

Common response (2/2)

Q4: Since JSON beats the text grid, the layout's own contribution is unclear; a non-grid JSON baseline would separate them. That chessboard beats images is expected, so its significance is unclear.

Our claim is not that layout works on its own. It is that layout and explicit structure add up. The chessboard and its JSON variant share the same spatial layout and differ only in serialization, and both beat the other encodings (Table 2). That is what adding up looks like.

Can the layout part be seen separately? Yes. In the within-family ablations (Table 3), widening the window to the full map or dropping the legend lowers the win rate in every comparison, and replacing raw coordinates with grid indices gains 0.3–1.5 points for the tree and the topology. The same structured information, written in coordinates that are easier to read, still helps.

On "expected": the minimap scoring below no encoding, topology falling to the baseline under the stronger model, and plain text paying for coordinate conversion are not outcomes one would predict in advance.

Q5: The setup is benchmark-specific: for other environments, models or RL policies the 4 s sweet spot could be 1.2 s or 6 s. A 30B LLM also seems heavy—do LLMs add anything over a rule-based planner?

The first contribution is a method. It turns a vague question, whether an LLM should do high-level planning at all, into two measurable design variables: how often to ask, and how to present the board within a given budget. It also gives the measurement recipe: sweep the interval against win rate, then compare encodings at a fixed interval.

Second, a full validation in one real game: 1v4 pursuit, seven maps, two models, eleven interval levels and seven encodings. The method runs end to end there and produces concrete magnitudes, 17.2 points from the interval and 7.8 from the encoding.

The procedure is not tied to this game. Any environment with real-time progression, state drift and a fixed low-level controller can follow it and get its own window and its own ranking. Four seconds is not a universal constant, and 3–5 s is only the flat region here; other games would calibrate their own numbers.

On whether an LLM is needed: the 41.2 baseline is a rule-based high level, a behavior tree, over the RL controller, with no model involved. Switching the high level to an LLM while keeping the low level fixed gives 49.6–62.8. The gap is the difference between a rule planner and an LLM planner.

Q6: Why use an LLM judge for staleness when many sensible definitions exist? Latency is nearly constant across intervals while the score drops sharply—how does the interval affect it, and which timestamps are used?

Start with how the interval is used. A decision is generated from the board at the start of an interval and then executed for the whole interval, until the next decision replaces it. Whether it matches is therefore judged against the board while it is in use, not at the instant it arrives.

The staleness score measures exactly that: how well the decision content matches the board it is executed against. A longer interval means a longer service window and more drift inside it, so the score falls, from 9.6 at 2 s to 1.6 at 25 s. Generation time stays near 1.2 s and occupies only a small part of that window, so the two are not in conflict.

Scoring is blind and at temperature zero, with three inputs: the board at decision time, the board at the reference time, and the decision itself. Scores are averaged per level. The score is used for ranking, not as an absolute. The judging LLM does not know the condition or the map, and its ranking matches the win-rate order measured by the engine, so we treat it as supporting evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.