- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of real-world games characterized by unknown, time-varying environmental parameters and critical information presented in unstructured formats, proposing the Context-Aware Game-Theoretic Agent (CAGTA). The framework leverages Large Language Models (LLMs) to convert contextual inputs—such as announcements and reports—into environmental parameters and their associated uncertainty ranges; it subsequently updates the model and replans whenever new information falls outside the scope of the current strategy. Theoretically, the authors analyze how effectively narrowing the uncertainty range enhances robust returns and derive a global error bound for multi-stage replanning under relevant assumptions. Experiments conducted on the GraphChase urban pursuit-evasion task—using LLMs to simultaneously handle interpretation and planning without additional training—evaluate performance across scenarios involving road closure information, time budgets, dynamic road weights, large-scale maps, and real-world urban announcements, demonstrating superior capture rates compared to selected baselines.
Critical strengths
The research addresses a problem of practical significance and employs a clear division of tasks within its framework. Focusing on how dynamic semantic information from real-world game scenarios is integrated into decision-making models, the paper proposes a workflow wherein an LLM interprets the context and updates environmental parameters, subsequently triggering a re-planning process. This design bridges language understanding and game planning, offering a clear pathway for adapting existing solution methods to environmental changes encountered during real-world deployment—a key aspect of the paper's practical value.
The paper provides a clear conditional analysis regarding the value of context. It demonstrates that effectively narrowing the scope of environmental uncertainty can improve worst-case payoff guarantees and further discusses the conditions under which performance in the actual environment is guaranteed to be no worse than that of the original strategy. This distinction avoids the simplistic assumption that "more information" automatically equates to "better actual performance"; it helps clarify the relationship between the accuracy of contextual interpretation and planning payoffs, thereby enhancing the method's interpretability.
Critical weaknesses
-
The comparison is not sufficiently fair to determine whether the performance gains stem from the proposed method itself or from the additional information provided. CAGTA not only received notifications of road closures but also had access to pre-calculated travel times and exit analyses; some prompts even explicitly instructed the model on which police officer to dispatch to which exit. In contrast, the Vanilla LLM lacked such assistance, and the large-map experiments primarily compared the method against baselines involving random movement or remaining stationary. Consequently, the current results fail to answer whether CAGTA would still outperform other planning methods if they were granted access to the same information, making it difficult for the paper to substantiate the independent contribution of its approach.
-
The testing scenarios are overly limited; a high capture rate is insufficient to demonstrate the method's reliability. The "100 paths" in the synthetic road-closure experiments actually consisted of a single path repeated 100 times, and the "20 paths" on real-world maps involved the same path played back repeatedly. Furthermore, the paper did not adequately test scenarios where the fugitive alters their route in response to police deployment or where the environment undergoes multiple consecutive changes. Thus, while the 100% capture rate confirms the system's success in a few fixed cases, it does not sufficiently support the claim that the system can handle diverse adversaries and continuously changing environments.
Questions for authors
CAGTA received pre-calculated travel times and deployment recommendations; did other methods also have access to this information? Has a comparative study been conducted where both parties were provided with the same information? If so, what further performance gains could CAGTA achieve?
Both road-blocking scenarios involved repeated testing on the same escape route. Beyond this, have existing experiments tested different escape routes, scenarios where the fugitive changes routes based on police deployment, or situations involving continuously changing road conditions? If so, please provide the corresponding results.
3 comments
Response
Thank you for the substantial revisions. The distinct-path evaluations and auction experiments address much of my concern about limited testing. However, the method’s added benefit remains unclear: CAGTA-LLM performs similarly to Vanilla-LLM under matched information in auctions, and the instruction-bearing vanilla baseline matches it in Toronto. These results do not sufficiently resolve my concern about gains beyond additional information and preprocessing, so I maintain my rating of 2 (weak rejection).
Respond to the new comments (reviewer o39w)
We thank the reviewer for stating the concern plainly. We answer it in steps, and we keep Table 3 unchanged.
1. The LLM planner is not the contribution
The paper's claim is the framework: an interpreter produces a game object, a solver plays it. Using LLM as the Role 2 is a heuristic instantiation; the exact solver is CAGTA+IGRS (and, in the auction, CAGTA+Shade).
2. The vanilla LLM is the control for the game model
The vanilla consumes the bulletin as text but does not map it into game parameters, as we mention in Section 5.1. The vanilla is therefore the control for the significance of the game model, not a competing planner.
3. Table 3, unchanged
- Blind: CAGTA+IGRS is 0.67; every vanilla row is 0. When the information is insufficient, the game model is what makes the decision.
- Informed: CAGTA+LLM, CAGTA+IGRS and the instruction-bearing vanilla all reach 1. When the instruction is enough, the game model is not necessary on this easy instance.
4. Control: when informed, the game model can still be necessary
To show that the game model can still be necessary in the informed case, we do a control experiments. That is, on a harder Toronto placement — outside officer near the gate, inside officer near E1, the true exit inside, a false exit outside, the same city bulletin — we run CAGTA+LLM and the two vanilla variants. No IGRS row; no late column.
| arm | blind | informed |
|---|---|---|
| CAGTA+LLM | 0/5 | 5/5 |
| Vanilla LLM (-) (instruction removed) | 0/5 | 0/5 |
| vanilla LLM (instruction-bearing) | 0/5 | 0/5 |
The geometry sets the trap: the outside officer’s believed route to E1 is 0.443 min through the shut gate, truth inf; the inside officer is the only true cover (2.841 min, criminal 3.352). The cover instruction sends the outside officer to E1 through the gate; CAGTA’s ETAs know that edge is inf and use the inside officer. So an instruction-bearing vanilla can fail even when informed, while CAGTA succeeds.
5. The auction, same structure
- The LLM planner is not the contribution; the solver rows are CAGTA+Shade.
- The vanilla is the control for the game model: it reads the notice and state but cannot produce Theta.
- Clean notices: Vanilla-LLM is close to CAGTA-LLM (0.111/0.039/0.014 against 0.121/0.044/0.012; Table 4, per-seed ranges in Table 8) — the text alone is enough there.
- The game model’s gain: CAGTA+Shade, given Theta, beats both in every regime (0.148/0.063/0.046; Table 4), while blind loses (0.121/0.022/−0.015).
- Control when informed: the mechanism-specific best response. The LLM bids near-truthfully (shading 0.96–0.98); the shader given Theta shades to 0.6 in first price (the calibration table in Appendix E gives the best gamma 1.00 in second price and 0.60 in first). Same notice and state, different game model: 0.063 against 0.039 per participation in first price with two bidders (Table 4).
Respond to Reviewer o39w
Thank you very much for your constructive comments.
W1 — the gain may be additional information, not the method. In Section 5.2 the six auction arms share the same numeric state and differ only in interpretation and solver (Table 4; Appendix E). We report the result as nuanced. After the rule change the bridge beats blindness (0.044 against 0.022 in first price with two bidders). CAGTA+Shade, given Theta, is stronger than the LLM. On these clean notices a Vanilla-LLM with the same text is competitive, so this table alone does not show that the interpreter is necessary. The bridge is visible where the prose has to be turned into parameters: on the Toronto bulletin, Vanilla LLM captures nothing in every arm, while CAGTA+LLM goes from 0 to 1 once the closure is read (Table 3).
W2 — limited scenarios. The revision adds the distinct-path IGRS values (25 / 95 / 193 paths; 0.5 vs 1.0; Table 2), adaptive competitors (Table 9), a four-regime day with three consecutive changes (Tables 10 and 11), and the second domain (Section 5.2). The paper states the route counts and the earlier repetition explicitly.
Q1 — fair information. Same comparison as in our reply to reviewer eaSX, Q1. With the same state in every arm (Section 5.2, Table 4): Vanilla-LLM scores 0.111 / 0.039 / 0.014 against CAGTA-LLM’s 0.121 / 0.044 / 0.012, inside the per-seed ranges (Table 8). On clean notices the bridge is not distinguishable from giving the text to the same LLM. The learner, which sees no notice, beats CAGTA-LLM before the changes (0.140 against 0.121) and falls below it afterwards (0.025 against 0.044; 0.009 against 0.012). CAGTA+Shade leads every regime (0.148 / 0.063 / 0.046). After the change, the ordering is solver-given-Theta > informed ~ vanilla-with-state > blind.
Q2 — diverse settings.