- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether LLM-based game agents can adjust their behavior according to graded natural-language priorities between playing strength and a requested play style. The authors evaluate Claude Sonnet 4.5, Gemini 2.5 Flash, and GPT-5 mini on Animal Shogi, Breakthrough, and Rush Hour. For each game, they construct positions where following the requested style conflicts with the game objective, and vary the relative priority of the two objectives using numerical, modal, and adverbial instructions.
Critical strengths
-
The paper is clearly written and the experimental setup is easy to follow. The distinction between playing strength and style adherence is well motivated, and the three instruction formulations are presented clearly.
-
The experimental design is careful. The use of exact solvers to construct positions where style-following genuinely conflicts with the game objective makes the main behavioral measurement easier to interpret.
-
The empirical evaluation includes a substantial number of robustness checks. In addition to the main three-game and three-model evaluation, the paper includes repeated sampling at fixed positions, competence conditioning, an open-weight model replication, instruction-formulation comparisons, position-selection sensitivity, and sequential Rush Hour experiments.
Critical weaknesses
W1. The practical significance of the observed style–strength trade-off is limited by the relatively low gameplay competence of the evaluated LLM agents.
The paper demonstrates that increasing style priority shifts action choices away from playing strength. However, the agents' performance ceiling is already fairly limited even when style is assigned zero priority. For example, at 0% style priority, win-retention rates in Animal Shogi are only 0.502, 0.403, and 0.549 for Claude Sonnet 4.5, Gemini 2.5 Flash, and GPT-5 mini, respectively, with even lower exact-optimal-action rates.
This makes the practical meaning of the resulting trade-off less clear. A useful controllable game agent would ideally start from a strong policy and then selectively sacrifice performance to satisfy a requested style. Prior work on strength adjustment has demonstrated this paradigm in much stronger game-playing systems, e.g., by adjusting strong MCTS-based Go agents toward designated human skill levels. In contrast, the present experiments primarily demonstrate controllability within a relatively low-competence regime. Although the competence-conditioned analysis shows that graded responsiveness is not solely caused by gameplay failures, it does not establish whether the same controllability remains meaningful for substantially stronger game-playing agents.
W2. The evaluation does not clearly establish what is uniquely gained by using an LLM rather than an existing preference-conditioned or strength-adjustable game agent.
Multi-objective and preference-conditioned game agents can already adapt their behavior according to desired objective weights, and prior work has demonstrated fine-grained strength adjustment in challenging games such as Go and chess. The main potential advantage of an LLM-based approach is therefore its ability to interpret flexible natural-language specifications without task-specific retraining.
However, the current evaluation uses a small set of predefined, programmatically measurable action-level styles and fixed instruction templates. These settings do not strongly exercise the open-ended linguistic flexibility that would distinguish an LLM-based agent from a conventional conditional policy. Experiments with unseen style descriptions, compositional preferences, or user-specified styles not explicitly anticipated by the evaluation would make the motivation for using an LLM considerably stronger.
W3. The priority grid is too coarse to support strong claims about smooth or fine-grained controllability.
The numerical experiments evaluate only five priority levels: 0%, 25%, 50%, 75%, and 100%. The reported curves connect these observed conditions but do not estimate a continuous response function. The results therefore convincingly establish an ordered or graded response, but they do not show that behavior varies smoothly or predictably between these widely spaced points.
This distinction is important if the intended contribution is fine-grained control over the style–strength trade-off. A much denser priority grid, ideally together with held-out intermediate values, would allow the authors to test whether the response genuinely interpolates between specified priorities rather than exhibiting threshold-like transitions that appear smooth only after connecting a small number of observations. This issue is even more pronounced for the modal and adverbial formulations, whose levels do not have a natural quantitative spacing.
W4. The action-parsing fallback may introduce an unquantified implementation bias into the single-decision results.
When no legal action identifier can be parsed from a model response, the implementation selects the first action in the supplied legal-action list. However, the reported validity audit only verifies that the stored actions are legal and does not separately report how often this fallback was triggered in the single-decision experiments. Therefore, the reported zero invalid-action rate does not rule out parser-induced changes to the measured action distribution.
This could matter if the ordering of legal actions is not randomized, since fallback events could systematically favor particular action types and thereby affect style-match or objective-retention rates. The paper should report the fallback frequency for each model and condition, clarify how legal actions are ordered, and ideally verify that the main results are unchanged when fallback cases are excluded or when legal-action order is randomized.
These are important becuase LLMs are not the best in the game domain. Therefore gaming can become both good benchmarks and diagnosis for the current mechanism. However, with limited ability, we are just not there yet to discuss the style of gaming and decrease the performance on LLMs. This is a quite centric problem when formulating the idea of the paper. I do think a better way to discuss is to at least mantain the ability while having different styles. I will consider the rebuttal but now I don't think this is really helpful for the community just yet. This paper is borderline between a score of 1 and 2. The writing itself is not that bad, but based on my current assessment, I lean more toward a 1. I will carefully consider the rebuttal and see how many of these concerns can be adequately addressed before deciding whether the score should be raised.
Questions for authors
Questions for Authors
-
How should the practical value of the demonstrated style–strength trade-off be interpreted given the relatively low gameplay competence at the strongest-performance endpoint?
Even at 0% style priority, the evaluated agents fail to preserve solver-established wins relatively often in some settings. Do the authors expect the same controllability to persist when starting from a substantially stronger game-playing policy, where reducing performance would represent a trade-off from a genuinely high-performance baseline? -
What is the main advantage of using an LLM for this problem compared with existing preference-conditioned or strength-adjustable game agents?
The current styles are predefined, simple, and programmatically measurable. If the intended advantage is zero-shot natural-language flexibility, could the authors evaluate unseen style descriptions, compositional preferences, or styles that were not explicitly anticipated when constructing the benchmark? -
Would the observed graded response remain smooth under a substantially denser numerical priority grid?
The current numerical evaluation uses only five levels: 0%, 25%, 50%, 75%, and 100%. These results establish an ordered response, but a coarse grid may hide threshold-like behavior between adjacent levels. Could the authors evaluate additional intermediate values, preferably including held-out priorities, to determine whether the behavior genuinely interpolates smoothly? -
Does the observed controllability extend to long-horizon competitive gameplay rather than primarily isolated decisions?
The complete-episode evaluation is performed only in Rush Hour, which is a single-player puzzle. Could the authors evaluate complete Animal Shogi or Breakthrough games to determine whether a requested style remains coherent across multiple decisions while interacting with an opponent? -
How representative are the conflict-selected positions of the situations encountered during ordinary gameplay?
The main evaluation deliberately selects states where every style-matching action conflicts with the game objective, while the appendix indicates that such conflicts occur in only a minority of style-available randomly sampled positions. Could the authors quantify the resulting style–strength trade-off on unconditioned gameplay states or full episodes, rather than primarily on states selected to maximize identifiability of the conflict?
6.How often was the first-legal-action fallback triggered in the single-decision experiments, and how were legal actions ordered in the prompt?
The current validity audit establishes that all stored actions are legal, but it does not separately identify parser fallback events. Could the authors report fallback rates by model and condition and verify that the main results remain unchanged after excluding such cases or randomizing the legal-action order?