acceptodds
Under review as a conference paper at ICLR 2027

How Well Do LLMs Adapt Their Play? A Controlled Evaluation with Classic, Misère, and Novel Board Games

Abstract

Board games provide a controlled setting for testing whether large language models adapt their behavior when task objectives or rules change. We study two forms of adaptation within a General Game Playing (GGP) framework. First, classic-misère pairs invert the success criterion while preserving the board, actions, legality rules, and state transitions. Second, Centralis, an original game designed for this study, introduces a new spatial structure, action semantics, and victory condition. All actions are validated by a common GDL-based referee. We evaluate Qwen3-8B and Qwen3-14B in Instruct and Reasoning configurations against four calibrated GGP reference opponents. The Reasoning configurations show fewer model-generated forfeits and greater recovery from invalid responses, but differences in competitive performance depend on the game and opponent and require substantially more tokens and execution time. The classic-misère comparisons reveal an objective-adaptation gap: win rates decrease or remain low in several pairings after the success criterion is inverted, despite unchanged game mechanics. In contrast, Qwen3-14B Reasoning achieves high win rates against all four reference opponents in Centralis, while Qwen3-8B Reasoning generally completes matches but records few wins. These results distinguish adaptation to an inverted objective from play under a new rule system and show that operational robustness, competitive performance, and inference cost characterize different aspects of model behavior.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, General Game Playing, Benchmarking, Rule Following, Natural Language, Game Rules, Strategic Adaptation, Inference-Time Compute, Move Legality, Competitive Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sjTDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether LLMs adapt when game objectives or rules change. It evaluates Qwen3-8B and Qwen3-14B on classic, misère, and novel board games using a shared GGP referee. Results suggest that reasoning improves action validity and recovery from errors, but does not consistently improve competitive performance.

Critical strengths

The common GDL referee provides a controlled evaluation environment with consistent legality checks and game semantics. Separating competitive performance from invalid-action recovery and inference cost is also sensible.

Critical weaknesses

Here are several weaknesses:

  1. The evaluation is limited to two models from the same family, neither of which represents the current state of the art. It covers only two sizes of Qwen3, two classic–misère game pairs, and one original game. As a result, the paper is neither sufficiently comprehensive as a benchmark nor sufficiently novel as a methodological contribution.
  1. The claimed “objective-adaptation gap” is not convincingly established. It is inferred mainly from lower win rates on the misère variants, but Table 1 shows that opponent strength also changes substantially across games.
  1. The comparison between Instruct and Reasoning is confounded by the generation budget: the former is limited to 128 tokens, while the latter receives 1,024 tokens. This is particularly problematic because many failures of Qwen3-14B Instruct are explicitly attributed to truncation.
Questions for authors
  1. Do the robustness differences remain when Instruct and Reasoning use the same generation budget?
  1. Can the authors provide direct evidence that the models choose moves appropriate for the classic objective but inappropriate for the inverted objective?
  1. Do the main findings hold for stronger models or models from other families?
2 comments
AuthorsOpenReview ↗

Response to Reviewer sjTD: Robustness, Move-Level Evidence, and Model Generalization

We thank the reviewer for highlighting important aspects regarding the interpretation and scope of the study. The questions help distinguish more clearly between the results already supported by the current protocol and those that could be further examined through additional controls or a broader set of evaluated models.

Q1. Do the robustness differences remain when Instruct and Reasoning use the same generation budget?

The study evaluates Instruct and Reasoning as complete inference configurations. Instruct is intended to produce direct responses without an explicit reasoning phase and therefore uses fewer tokens, whereas Reasoning is given a larger budget to support that phase. This setup allows both modes to be compared under conditions consistent with their intended use. An additional analysis with matched generation budgets could complement these results by specifically controlling the maximum generation length. Similarly, the experiments could be repeated using the provider-recommended configuration for each mode while keeping the rest of the protocol unchanged. In addition, Qwen3-8B Instruct and Qwen3-14B Instruct use exactly the same configuration and still show clearly different robustness profiles. This indicates that the observed behavior does not depend only on the generation budget.

Q2. Can the authors provide direct evidence that the models choose moves appropriate for the classic objective but inappropriate for the inverted objective?

The current results mainly capture behavioral changes at the match level. The classic–misère pairs preserve the board, action space, legality rules, and state transitions while changing the success criterion, but a reduction in win rate alone does not identify which specific decisions produce that difference. The logs allow us to examine this point directly. We can analyze missed immediate winning opportunities in TicTacToe/Connect4, avoidable completions of the model’s own lines in Suicide/NotConnect4, and the proportion of actions consistent with the active objective. This would provide move-level evidence of a possible carry-over from the classic objective and complement the aggregate results without attributing the observed behavior to a specific causal mechanism.

Q3. Do the main findings hold for stronger models or models from other families?

The current study is not intended to establish broad generalization across model families. We used two Qwen3 sizes to maintain a controlled experimental setting and compare configurations under the same protocol. However, we have conducted preliminary tests with additional models. In the misère games, we also observed a decrease in performance relative to their corresponding classic variants. In Centralis, at least one of the frontier models tested achieves a high win rate, showing that a model outside the main experimental configurations can understand and play the new rule system effectively. These results are preliminary and are not used to support the current conclusions. Including at least one frontier model in an extended version would allow us to examine more directly which patterns persist beyond the Qwen3 family and which may depend on model capability or inference configuration.

Thank you for your response and for the effort you put into addressing my concerns. I believe a benchmark should offer insight beyond reporting which model achieves which score on each task. The results presented here give the community little guidance on why certain tasks remain difficult or where follow-up work would be most useful. The evaluation also covers only two models, which makes it difficult to tell whether the findings generalize across approaches. For these reasons, I will maintain my original score.

Reviewer_PxhbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates how Qwen3 models play classic games, variants with inverted winning conditions, and a newly designed game, Centralis. A shared GGP referee supports comparisons of competitive outcomes, invalid moves, recovery, and inference cost.

Critical strengths

The classic misère pairs provide a useful setting for studying objective changes while preserving the board and action interface, and Centralis adds an unfamiliar rule system. The evaluation balances player order, calibrates reference opponents, and reports competitive performance separately from operational failures and cost.

Critical weaknesses
  1. Lower win rates in misère games do not by themselves establish an adaptation deficit, because objective inversion also changes strategic difficulty and the effective strength of each opponent.
  2. The 128-token limit causes frequent truncation-related forfeits in the Qwen3-14B Instruct condition. These failures make it difficult to assess the model’s ability to adapt its strategy to different game objectives.
Questions for authors
  1. Can a move-level analysis on matched board states distinguish failure to follow the inverted objective from differences in strategic difficulty?
  2. Does Qwen3-14B Instruct show the same adaptation gap when the output budget is increased enough to make truncation-related forfeits rare?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Pxhb: Objective Inversion and Inference Configuration

We thank the reviewer for the constructive feedback and for recognizing the strengths of the experimental design, including the classic–misère comparison, Centralis, balanced player order, calibrated GGP reference opponents, and the separate analysis of performance, robustness, and cost.

Q1. Can move-level analysis distinguish failure to follow the inverted objective from differences in strategic difficulty?

We agree that a move-level analysis would provide more direct evidence. The purpose of the classic–misère pairs is to preserve the board, action space, legality rules, and state transitions while changing only the success criterion, allowing us to examine behavioral changes under objective inversion. At the same time, we acknowledge that strategic difficulty may also change, and the manuscript already avoids attributing the observed gap to a specific causal mechanism. We will make this distinction clearer and consider matched-state analysis as an additional analysis using the move-level logs already generated by our protocol.

Q2. Does Qwen3-14B Instruct show the same behavior with a larger output budget?

The Instruct configuration is designed to produce direct responses without an explicit reasoning phase, so its token consumption is naturally lower than that of Reasoning. For this reason, we used a maximum of 128 tokens per call for Instruct, compared with 1,024 for Reasoning. Even so, we acknowledge that increasing the budget for Qwen3-14B Instruct would help determine how much of its behavior is related to truncation. It is worth noting that Qwen3-8B Instruct and Qwen3-14B Instruct were evaluated under exactly the same configuration, and while the former completes almost all classic–misère matches, the latter produces many more truncated responses. This suggests that the observed behavior is not simply due to the two models receiving different generation budgets. As an additional control, the experiments could be repeated using the provider-recommended configuration for each mode, while keeping the rest of the experimental protocol unchanged.

I am keeping my original score. In the classic–misère comparison, the board and legal moves stay the same, but reversing the objective can change how difficult the game is and how strong the reference opponents are. That leaves the lower win rates open to more than one interpretation. The Qwen3-8B comparison shows that the two Instruct models handle the same 128-token budget differently. For Qwen3-14B, however, frequent truncation-related forfeits still make it difficult to tell how well it adjusts its play when it can return a complete move.

Reviewer_jgx8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates Qwen3-8B and Qwen3-14B, each in an instruct (128-token budget, no reasoning) and a reasoning (1,024-token budget, low effort) configuration, on five GDL games under a single Gamemaster referee: two classic/misère pairs (TicTacToe/Suicide, Connect4/NotConnect4) that keep the mechanics and invert the win condition, and Centralis, a new radial game. Each condition plays 60 order-balanced matches per game against legal, random, greedy and mcs agents (4,800 matches), with ten invalid responses allowed before forfeit. A GGP round-robin calibrates the opponents (Table 1). The headline claims are an "objective-adaptation gap" after inversion, fewer forfeits under reasoning at higher cost, and 14B reasoning winning 0.73 to 0.93 of Centralis matches.

Overall: The harness and the misère-pair idea are sound, but the headline gap is not distinguished from a change in opponent difficulty while the move-level analysis that could settle it sits unused, the token counts contradict the stated configuration, the 14B instruct condition needs to be shown to be a model property rather than a truncation artefact, and the intervals rest on an independence not demonstrated. A corrected version with the matched-position analysis could be a reasonable evaluation paper; this one is not ready.

Critical strengths

The harness is carefully built and reusable: one GDL referee adjudicates legality and termination for every agent (Secs. 3, 5.2), order is balanced, infrastructure failures are excluded rather than counted as forfeits (Sec. 5.3), and invalid responses are decomposed by cause (Fig. 2). The artefact keeps its value even where the behavioural conclusions fail, and the analyses below can be run on logs the harness already produces.

Critical weaknesses
  1. The "objective-adaptation gap" is not shown to be about adaptation. Inverting the objective changes the strategic problem and the opponents' play with no LLM involved (Table 1: Random beats Legal 0.14 in Connect4 but 0.85 in NotConnect4; Legal beats Random 0.57 in TicTacToe but 0.37 in Suicide), so a drop in Table 2 is equally consistent with "misère is a different game against these opponents". Secs. 7 and 8 concede that carry-over, difficulty and prompt salience remain compatible explanations, yet the abstract presents the gap as a finding, and instruct cells move the other way (8B instruct vs legal: 17 wins in Connect4, 53 in NotConnect4). The direct test is cheap and sits in the move logs: in matched TicTacToe/Suicide positions, how often does the model pick a move that wins classically but loses under misère when a safe alternative exists? Without it, the central claim is a descriptive cross-game difference, and one novel game does not establish adaptation to novelty either.

  2. The executed configuration does not match the described one, so the condition contrasts cannot be attributed. Sec. 4.1 caps reasoning at 1,024 tokens per call, yet Table 8 reports 5,534 completion tokens per match for 8B reasoning vs legal in TicTacToe with 0% retries (Fig. 2); five LLM moves at most bound that match at 5,120, so either the cap is not a per-call total or something else is counted. The Centralis cell (194,482) depends on whether the ten retries are per move or per match, which the paper does not say. Separately, 14B instruct shows 2.9 to 5.7 format errors per turn in every game while 8B instruct shows 0.00 under the same 128-token limit (Fig. 2), and emits 1,022 to 2,387 tokens per match against 18 to 177 for 8B (Table 8). That pattern is consistent with reasoning text still being generated and truncated on every call, which would make a quarter of Table 2 (11 to 60 forfeits per cell) a deployment artefact; Sec. 7 instead reads it as model behaviour without showing a raw response. The paper needs backend identifiers, the actual thinking-mode setting applied, and a sample of raw 14B outputs. Also, Legal is deterministic (Table 6) and decoding is temperature 0 with a fixed seed (Sec. 4.1), so the 30 matches per order against legal may be few distinct trajectories (the 30/0/30/0 cells in Table 2 and the sd 0 token cell in Table 8 suggest so); the n = 60 Wilson intervals in Fig. 1 assume an independence that has to be demonstrated with unique-trajectory counts.

Other issues: win rate ignores draws (14B reasoning vs greedy, TicTacToe: 0/43/17); no split by player order despite first-mover effects near 0.5 (Fig. 5b); instruct vs reasoning confounds effort with an 8x token budget; 14B reasoning beating mcs 50/0/10 in Centralis is unexplained though mcs beats greedy only 0.14 there; prompts are not reproduced.

Presentation: Readable, but the Appendix C heading on p.15 points at nothing (the cost results are Table 8 on p.17), Table 3 has an untranslated header, Table 5 repeats 98/53/49 in both Greedy-vs-MCS Suicide cells where wins and losses should swap, and the Centralis win paths in the Fig. 3 label contradict Sec. 3, so I am not sure which rules the models saw.

Questions for authors
  1. [suggestion] From the move logs: per condition and role, the fraction of chosen moves that are optimal under the active objective, and the rate of avoidable own-line completions in Suicide/NotConnect4 vs missed immediate wins in TicTacToe/Connect4. This would directly test carry-over and could change my assessment.
  2. [clarification] How can Table 8 report 5,534 completion tokens for a no-retry TicTacToe match under a 1,024-token cap? Is the retry budget per move or per match?
  3. [clarification] Did raw 14B instruct responses contain reasoning text before truncation? Which OpenRouter parameters and providers served each model? A budget-matched control (14B instruct at 1,024 tokens) would separate truncation from reasoning.
  4. [clarification] How many distinct move sequences occur among the 30 matches per order in each cell against legal?
  5. [clarification] Which Centralis rules were in the prompt, Sec. 3's or Fig. 3's? Please include all five prompts verbatim, and check Table 5's duplicated Greedy/MCS Suicide cells against the ledger.
  6. [suggestion] Compare LLM win rates with the calibration floor: 8B reasoning wins 0.42 vs legal in Suicide where Random wins 0.60, and 14B reasoning 0.47 in NotConnect4 where Random wins 0.85, while both far exceed Random in the classic games. Below-random play after inversion supports your thesis better than raw drops do.
1 comment
AuthorsOpenReview ↗

Response to Reviewer jgx8: Objective Adaptation, Experimental Configuration, and Additional Analyses

We thank the reviewer for the detailed reading and for recognizing the experimental infrastructure, the common GDL-based referee, and the value of the protocol and artifacts. The comments help us refine the interpretation of the results and strengthen the analysis.

S1. Move-level analysis to evaluate adaptation to the objective

The classic–misère pairs preserve the board, action space, legality rules, and state transitions while changing only the success criterion. The existing move-level logs already contain the information needed for the suggested analysis. We can analyze, by condition and role, actions consistent with the active objective, missed immediate wins in TicTacToe/Connect4, and avoidable own-line completions in Suicide/NotConnect4. This provides a more direct test of possible carry-over from the classic objective while keeping the "objective-adaptation gap" interpretation behavioral rather than causal.

Q1. How can Table 8 report 5,534 completion tokens with a maximum of 1,024? Is the retry budget per move or per match?

The 1,024-token limit applies to each individual model call, whereas Table 8 reports completion tokens accumulated over the entire match. A match therefore contains multiple calls even without retries. The retry budget applies per move: an invalid response can trigger additional calls during the same turn. We will clarify both points in the protocol and Table 8 caption.

Q2. Qwen3-14B Instruct, truncation, and OpenRouter configuration

Instruct and Reasoning were evaluated as complete inference configurations. Instruct uses 128 tokens per call with "reasoning_effort=none", while Reasoning uses 1,024 with "reasoning_effort=low". The lower Instruct budget reflects its intended use for direct responses without a prolonged explicit reasoning phase.

Qwen3-8B and Qwen3-14B Instruct use exactly the same configuration, yet 8B completes almost all classic–misère matches while 14B shows substantially more truncation-related responses. Thus, their difference cannot be explained by unequal generation budgets.

As an additional control, we can repeat the experiments using the provider-recommended configurations for Instruct and Reasoning. We will also provide the exact OpenRouter identifiers, inference parameters, and representative raw responses associated with truncation.

Q3. How many distinct sequences occur among the 30 matches per order against Legal?

Legal is deterministic, while the LLM uses temperature zero and a fixed seed, so trajectories may repeat when the same states occur. Random and MCS instead use different seeds between matches and can generate different trajectories. We can report unique trajectories by condition and role against Legal and qualify the confidence intervals where appropriate.

Q4. Centralis rules, prompts, and verification of Table 5

The experimental prompt is the one defined in the execution code. It specifies the initially locked center and the radial, clockwise diagonal, and counter-clockwise diagonal paths required to unlock it, followed by center occupation as the winning condition.

We will align the description in Figure 3 with the implemented GDL rules and actual prompt, include all five prompts verbatim in the reproducibility material, and compare the Greedy/MCS Suicide values in Table 5 with the original logs to confirm consistency.

S2. Comparison with the Random reference level

We appreciate this suggestion. GGP performance varies by game, so we do not treat the agents as a fixed difficulty hierarchy. However, Random provides a useful within-domain reference.

As the reviewer notes, Qwen3-8B Reasoning scores 0.42 against Legal in Suicide versus 0.60 for Random, while Qwen3-14B Reasoning scores 0.47 in NotConnect4 versus 0.85 for Random. In the corresponding classic games, both models clearly outperform Random against the same opponent. Thus, in some conditions, objective inversion not only reduces performance relative to the classic game but places the model below a calibrated Random agent in the same domain. This complements the move-level analysis proposed in S1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.